Christopher C. Wells

Reading the 2025 Hugo Finalists

2025-04-18T08:30:00-07:00

Having finished reading through the past 71 years of Hugo award winners and with the 2025 Hugo Award finalists announced, I'm planning to spend the next few months reading some of the 2025 finalists

I won't be writing "reviews" of the novels by any means. I will simply be capturing my opinions about each one so I can think about them as a whole during voting. I hope to keep this up year-to-year as I am a somewhat slow reader and I can't even imagine trying to follow more novels over the course of the year in order to actually nominate something.

Best Novel

Best Novel Vote

Someone You Can Build a Nest In by John Wiswell
Alien Clay by Adrian Tchaikovsky
The Tainted Cup by Robert Jackson Bennett
A Sorceress Comes to Call by T. Kingfisher
Service Model by Adrian Tchaikovsky
The Ministry of Time by Kaliane Bradely

Someone You Can Build a Nest In

Someone You Can Build a Nest In by John Wiswell deals with themes that most other Hugo winners I have read do not (or do little): dependence, pain, family, parenting, "otherness", self. It is a challenging read, yet has some good dark humor and a lot of action and adventure packed in.

The opening paragraphs set up the world of the novel splendidly and makes one eager to read more (quickly, in my case). From her very birth, in memory, of Shesheshen to her hibernation and immediate peril there is action and interest in her experiences -- I felt on "her side" right away and carried that relationship through the whole novel.

Then the introduction of Homily and slow unrolling of her life and relationship with Shesheshen made for an amazing story that was times silly and other times difficult to read and face. Homily's dependence on her family and Shesheshen's confusion over her feelings about Homily mixed so well even while keeping a pretty fast action pace with the Wulfyre's hunt for the wyrm.

The ending was satisfying and my only real knock on the story was in the penultimate Part Seven. Shesheshen's confusing (even in the state she was in) with her sister felt a little unbelievable and predictable as it was playing out. And to some degree I lost the path of the Baroness in this section -- she seemed to kind of fade away inexplicably (OK, so a bomb had something to do with it) and then reappear in a much less favorable position for herself. That is, after her encounter with Shesheshen I would've expected her to seek out Epigram and Homily instead of allowing Homily and Shesheshen to connect again.

Wiswell's dry, dark humor in places reminded me of Connie Willis and Martha Wells. And reading the acknowledgments at the end made it clear that Wiswell poured a lot of himself and his experiences in to the novel. That was surely no small task and I'm thankful he was able to make it work so well.

Service Model

Reading Service Model by Adrian Tchaikovsky was neither good nor bad. It was just something I did. I was excited going in to this one given some favorable reviews comparing the writing to John Scalzi and Martha Wells (for Murderbot). The first few chapters dealing with Charles's crime, the doctor's visit, and the investigation of the detective was silly and downright hilarious at parts. From there the novel felt repetitive and tedious as Uncharles traveled along with "the Wonk" and learned about the post-apocalyptic state of the world.

My mental image through most of the novel used characters straight out of Futurama. In my teens and twenties I was a huge fan of Futurama and a lot of the situations and scenarios Unchales and the Wonk encounter would have fit perfectly in to any season. From silly logical quips to obsessions with efficiency, the ridiculousness of bureaucracy, and the non-perfect functioning of systems that are meant to make human lives easier Service Model pokes fun at many of the same things that Futurama did. In this respect, Service Model was hindered by a much smaller "main cast" in Uncharles and the Wonk. Other characters came and went, but none stuck around or made much of impression (except perhaps the short-lived Inspector Birdbot and Doctor Namehere) as the giant cast of Futurama.

After that, the farm, library, wasteland, and courthouse were more serious, less silly, and not well enough developed to really get in to. The dark humor lost a lot of it's edge as things got more serious along the way and none of the other secondary characters stood out as much. I also could never quite place how I felt about Uncharles not being unable to recognize that the Wonk was human. This led to a lot of amusing little anecdotes and did build up nicely in the end, but it was clear enough to the reader very early on that the revelation felt somewhat hollow and uninteresting when it (finally) happened.

The Tainted Cup

The Tainted Cup by Robert Jackson Bennett is just as massive as the leviathans it portrays, which are only a secondary aspect of the plot! The scope of the novel reminded me of the first time I read The Fifth Season and I suspect that the next book in Bennett's planned series will show up as a Hugo finalist next year as well (much like the Broken Earth series). One key difference that I like about The Tainted Cup is the the first person POV (as opposed to Broken Earth's third person POV). I assume this is more common for murder mysteries anyway (I don't read many), but it helped pull me deeply in to the mystery and share confusion and excitement with Din (who is an excellent storyteller).

The first few chapters were pretty heavy on the world-building and that paid off as the murders piled up and the plot thicken. The background of the leviathans and the forced deadline for Din and Ana's work was just as intriguing as the main storyline. I hope top read more in the series in the future.

Once the background was establish I settled in to Din's voice and perspective. The unveiling of his special abilities and his fears about his shortcomings is well paced to the events of the novel. His relationship with Strovi could have used more development and felt a bit "tacked on", but presumably (and hopefully?) that will built upon somehow in upcoming novels in the series (if not with Strovi specifically). Din's relationship with Miljin was much better addressed given it's important to the story and although it seems unlikely, I'd love to see more of Miljin in the future.

Ana's eccentricities are often silly and fun to read. Her Sherlockian attributes are clear even at the very end of the novel when Din gifts her a box of "moodies", making her happy as can be. Bennett does an excellent job providing the clues and pacing her revelations about each of the various murders. I recalled wondering at one moment about late in the novel what the title of has to do with anything only to have it revealed within a couple of pages. This felt random, but was actually well set up by the development of the plot and the big reveals in the final few chapters.

The Ministry of Time

The Ministry of Time by Kaliane Bradely imagines a depressing future where time travel is possible and the furthest known future is only a few hundred years away and in dire straits. So much so that the perceived "villains" of the first half of the novel end up seeming to me like the most redeemable characters in the novel. It does not end well for them in the novel's present or future and mostly I just felt bad for them by the end of the book.

To Bradely's credit, the narrator is an excellent example of a unreliable narrator. I spent much of the novel not trusting her and not liking her. The last few chapters just made that dislike even more intense -- to the point that I didn't much care about her (or the main plot) at all. In a way, that's an impressive achievement. Unfortunately for me, it made reading the novel quite unenjoyable.

In the end, as the narrator continues to selfishly pursue her singular goal with no regard for the potentially devastating side effects, I mostly felt I had read a cautionary tale about the climate future of our planet. It felt real and depressing. I need something to anchor on in the novel and it just wasn't there. The expats were certainly likeable and fun to read at times, but ultimately they were only props to the inevitable storyline that never really piqued my interest.

Aside from the time travel aspect, I struggled to think of this even as speculative fiction. It felt more like a very slow burning spy thriller that just happened to include some time travel (i.e., most of the story could have remained roughly the same without anyone walking through a time door).

A Sorceress Comes to Call

A Sorceress Comes to Call by T. Kingfisher is a character-driven story of deception and sorcery that builds slowly as all the pieces come together for an intense and violent finale. Most of the characters are well developed, there is lots of dark humor peppered throughout, and by the end I felt invested in the outcome.

Cordelia's difficult relationship with her mother is established quickly in the first few chapters, but feels like the least developed relationship overall -- Evangeline is terrible to Cordelia and seems to see only her as an extension of herself. That doesn't change much throughout the novel. Cordelia is extensively developed through her relationships with all the other characters while Evangeline seems to remain mostly static.

Hester and Lord Evermore's romantic subplot was, by comparison, deeply complex and well developed. I often found myself more interested in the development of that relationship than what was happening with Cordelia and Evangeline, possibly just because of the slow pacing of the novel overall.

The various secondary characters were fun to read. Alice and Cordelia's relationship felt so vital to Alice's development. And Willard delivered some absolutely hilarious lines in the last stretch of the novel.

Overall, I felt left just wanting to know more; about Evangeline primarily. But also about Falada, and Hester and Lord Evermore before Cordelia came into their lives. This novel read to me like a portion of a larger work and the slow development made it hard to just sink in to and read quacky.

Alien Clay

Alien Clay by Adrian Tchaikovsky is exciting mix of broad, creative sci-fi, black humor, and political intrigue that perhaps feels a bit too close to current day politics. From the very beginning I am taken by Daghdev's plight and ridiculous ability to survive his arrival on Kiln and unfold the story of how he was sent there. The combination of a vast, highly imaginative sci-fi world and a fast-paced political action plot is something I am always drawn to. Tchaikovsky is an excellent in these regards.

Daghdev's constant foreshadowing and examination of the situation and environment on Kiln is engaging for it's mix of serious and silly elements. I found myself really wanting the first revolt to succeed even though it was pretty clear it would fail. Tchaikovsky's exploration of how and why movements against entrenched fascist regimes ultimately fail feels accurate (and frustrating!).

Tying this thesis to Kiln's very nature and the second revolt is expertly done and makes for an enjoyable, but also deep read. It was hard to put down in the final chapters and is one of the better novel endings I have read in a while for how well it wraps up all the loose ends and provides an tantalizing setup for the character's next steps (even without necessarily making it feel like a series is needed).

Best Novella

The Tusks of Extinction by Ray Nayler: a this story fits wonderfully in the "novella" category. There is just enough story, backstory, character development, and action. I'm not left wanting to know more or feeling like I didn't know enough. The plot is creative, the characters are engaging, and the slightly non-linear storytelling works well for the story. Damira's experience felt both sufficiently "sci-fi" and seemingly like a potentially reasonable thing that could in a not-too-distant future. I highly enjoy this type of speculative fiction.
What Feasts at Night by T. Kingfisher: a dark, haunting fairy tale about a possible witch in a cabin in the woods. Excellent stuff. I was not able to put this one down for the last third of the book. I particularly loved Kingfisher's treatment of silence as a heavy, terrifying thing. I'll have to grab What Moves the Dead for a future read.
The Butcher of the Forest by Premee Mohamed: a tantalizing preview of a much larger world. At times this felt similar to Nettle & Bone in it's fairy tale quality and storytelling. Veris's "false" worst memory was more graphic than it needed to be. And the Tyrant's decision about her fate in the end didn't fit well overall. This is story that would have benefitted more from a dark, depressing unhappy ending (not that it wasn't, it just could've been worse).
The Practice, the Horizon, and the Chain by Sofia Samatar: an enjoyable read, but lacked any major "hook" in the story. Manual mining of asteroids using slave labor seems quite far-fetched. I was hoping for some twist that explained more about why the imagined society had established these casts beyond just: "to mine minerals".
Navigational Entanglements by Aliette de Bodard: a lot of good elements in this story that added up to just too much content for a novella. I needed more time to understand the clan and personality relationships and also to envision what a "shadow" is. The actions and development of characters just came too fast and had too much going on for me to settle in to the story and/or enjoy any particular aspect of it.
The Brides of High Hill by Nghi Vo: while I enjoyed the way the novel ended, I felt like I missing a lot of pieces about the main character that presumably would have come from the other novellas in the series. It was difficult to connect with and understand Chih and too much of what unfolded in the story seemed require that understanding. I would rate this higher if considering an award specific to horror.

Best Dramatic Presentation (Long Form)

Dune 2: I was a big fan of the first film in this series and the sequel absolutely did not disappoint. Excellent acting, cinematography, music, sounds, story. All around epic quality film.
Flow: I watched this with young children (ages five and seven). I was worried they may not enjoy the different style of animal movie, but they were very into and loved talking about the animals and what they thought was happening in the world of the film. It was fun to enjoy a different style of animal movies for younger audiences.
Wicked: I have never seen the musical so the movie was entirely new content and story for me. I enjoyed the story and the singing was excellent. My biggest complaint is probably just that it's only a Part One. It felt pretty complete as-is for setting up the character.
The Wild Robot: This one was also popular with young children. The story and plot felt a little more interesting and in depth and I enjoyed the animation and voice performances.
Furiosa: A Mad Max Saga: I was hoping for a Mad Max sequel more than this, but I enjoyed it more than I thought I would. The story and acting was good. My major complaint is with the odd style and cinematography used extensively in the film. Something felt "fake" or overly edited about some of the scenes and it was hard to shake that.
I Saw the TV Glow: There was a lot of promise in this film. I liked the concept, the shooting style, the music, and the characters. The presentation was just too disjointed to settle in to. I wanted more Pink Opaque content and a deeper exploration of Maddy's beliefs about what was going on. And the end was just not satisfy enough for all of the possibilities that the film set up. Late 90s teen awkwardness was captures expertly so mostly I came away from the film feeling nostalgic and depressed.

Reading 71 Years of Hugo Award Winners

2025-04-17T20:40:00-07:00

Sometime back in May 2019 I started reading award winners of the Hugo award for best novel in chronological order starting with the first novel awarded (from 1953).

I regret not taking more time to document the process. I don't even recall how I decided to follow the list. I just needed something to help me to decide what to read at all. May 2019 is my best estimate based on exchange I had with my wife on Telegram in early June about Ben Reich from The Demolished Man. Beyond that, I have kept only minimal "ratings" of the books and whatever stuck with me through 5+ years.

I finished up just in time for the announcement of the 2025 Hugo award finalists so now I'm going to take the time to read all the nominees for the year and try to do a better job documenting my thoughts.

Here are some short thoughts and recollections from all that reading (not reviews by any means!).

1950s and 60s

My primary memory of the first 17 years of books is The Man in the High Castle. My only exposure to Philip K. Dick before reading this one was his influence in film: Blade Runner, Total Recall, Screamers, Minority Report, A Scanner Darkly, etc. The Man in the High Castle is the earliest of the reads with a more "believable" alternate reality. The scene that really sticks with me is Tagomi's "spiritual experience" of the alternate reality where Germany and Japan did not win World War II.

Outside of that I most enjoyed Starship Troopers, The Wanderer, Dune, and The Moon Is a Harsh Mistress.

There were two I did not enjoy at all: The Big Time, which felt more like reading a play than a novel, and Stand on Zanzibar, which I did not actually finish. I'd like to at least go back to Zanzibar someday as I don't recall what I disliked about it.

1970s

Dreamsnake dominates almost all my memories of this period. It was the best novel I read to date and remains one of my favorites of all the Hugo winners. McIntyre's development of Snake as the main character is masterful and this felt like the most developed of any female characters among Hugo winners to that point by far. If one read Dreamsnake out of the blue today, the impact may not be as great. But reading it with the context of all previous Hugo winners, it felt like something entirely different and well ahead of it's time.

The other novels from this era I enjoyed were: Ringworld, Rendezvous with Rama and The Forever War.

1980s

Just being able to produce a novel about dolphins in spaceships is an achievement in itself. Then to make the story (and ultimately the series) compelling and entertaining is just something else entirely. I remember going in to Startide Rising with a lot of skepticism but Brin builds an amazingly creative and interesting world.

The concept of uplift in general would benefit from a lot more literature and there are other novels I'll check out in the future that deal with it. The Uplift War was also a fun read and maybe only doesn't register as a favorite because Startide was some good and I read it first. I'll probably read more of Brin's series in the future (when I have decide on stuff to read on my own!).

The thing I enjoyed most about Ender's Game is that I went in to it with no background about the book or movie. The build up to the midpoint of the novel was entertaining and I didn't really expect it to develop in the way it did for the second half. The second half, Ender's experience with the queen and quest to find a new home for the "buggers", was what really drew me in and I finished this one excited to read the sequel next.

In general reading through the Hugo's in order has been kind of difficult when I come across a series or author I enjoy. I wanted to stick to the plan and avoiding reading anything else (also because I am a slow reader and wanted to actually finish/catch up the present) so getting to read a sequel right away was a treat.

I remember thinking of Speaker for the Dead as the better of the two novels. It felt like a much needed continuation of Ender's Game to explore the more interesting themes dealing with Ender's role as "speaker for the dead".

1990s

The years of Lois McMaster Bujold and Connie Willis!

The Vor Game, Barrayar, and Mirror Dance are all such fun reads! The development of Miles as a primary character that uses intellect over strength in a universe that highly prioritizes physical attributes like strength and beauty is excellent. Bujold builds good suspense and action, keeps plots moving, and creates compelling characters and storylines. While I have stayed mostly committed to the list, this is definitely a universe that I will revisit (and probably push my young children too when they are more in the "Young Adult" reading age).

1993 was a great year for the Hugo award. I thoroughly enjoyed A Fire Upon the Deep and it only suffers in memory because of how much I loved Doomsday Book. I remember feeling intimidated by Doomsday Book for it's length but ended up reading it quickly because it was just so intense and ridiculous and enjoyable. I also ending up reading and learning a lot about the Black Death and just became so attached to Kivrin's plight and the efforts to recover her. It is one of the most enthralling, terrifying, and humorous books I've ever read.

2000s

The two winners from the 2000s that stand out most in my memory are Paladin of Souls and The Yiddish Policemen's Union.

Paladin felt by far the most like a "fantasy" as opposed to sci-fi and despite my enjoyment of everything else Bujold I was a skeptical going in it. But of course the story was complex and interesting, the characters were well developed, and I read through it much faster than usual because it was so engrossing.

The Yiddish Policemen's Union reminded me of The Man in the High Castle as they both created well defined alternative realities that were easy to accept and enjoy reading about. This was also the first detective novel I have read and I found I thoroughly enjoyed it.

2010s

The 2010s were full of great novels, or maybe it's just the fact that I remember more clearly those recent reads.

Just a few years after The Yiddish Policemen's Union I ended up enjoying another detective novel in The City and the City. This was my first exposure to weird fiction and its definitely a genre I'll pursue more in the future. I also sought out and enjoyed the BBC Two TV series adaptation and was happy with how it compared to what I saw in mind (mostly).

I have loved everything Connie Willis and was really looking forward to Blackout/All Clear. Her style and substance was there but clocking in at over 1,000 pages it did feel a bit like a slog towards the end (and was very clearly meant to be one novel given how abruptly Blackout ended). By no means did it turn me off to Willis, it just felt like a lot.

Comparatively, Among Others was a fun, short read. It felt close to weird fiction, but maybe that was because I had read The City and the City so recently. I really loved the way the story hinted at magical things without committing and keeping something of an unreliable narrator vibe. It genuinely felt like I was reading a 15 year old's diary.

I went in to Redshirts without knowing anything about the novel or John Scalzi generally and boy was it a fun surprise. MY only exposure to Star Trek has been the recent films so I got to spend lots of times looking up and learning about the ridiculous and fun little things the novel poked fun at. It was nice to be read something so silly.

Reading Ancillary Justice after all the more genre-twisting entries from the 2000s and 2010s may have impacted my feelings about it. While I enjoyed the read, it felt more "old school" in a bad way and perhaps a bit boring. There are many entries from the earlier years I do not mention here for similar reasons -- they were OK, if not memorable.

The Three-Body Problem was certainly the most "hyped" novel I read of any of the Hugo winners. It was the only one I'd heard of outside the context of reading through the Hugos. I went in to thinking it must be overrated, but I was very much (and happily) wrong! It is a great sci-fi/kind of horror mix. I watched and mostly enjoyed the Netflix adaptation and someday I'll commit the time to the Chinese TV adaptation.

I absolutely loved the Broken Earth series. Just a lovely blend of sci-fi and fantasy and ultimately an insane, difficult, and touching story about motherhood. Since becoming a parent I tend to find it difficult to read books and watch TV or film that deals with parenting (or young children generally) and this series does a great job of exploring the difficulties of those relationships.

2020s (so far)

A Memory Called Empire and A Desolation Called Peace felt like sort of anti-Ancillary Justice in that they brought more depth and intensity to the space opera and kind of "updated" it over their predecessors. I enjoyed the "political" aspects of the novel and how it examined language and culture. It reminded me about how much moving out of the United States for a few years got me to recognize the intricacies of culture.

Network Effect was such a joy to read. Like Redshirts, it brought some fun in to the genre and Wells's writing and humor is just wonderful. Her dry and sometimes sadistic humor reminded me of how I felt early in my life reading Vonnegut. After so many years of sticking to reading the Hugo award winners and nothing else I actually took a short break to read up everything The Murderbot Diaries and I was sad when I finished them all. I've got high hopes for the Apple TV adaptation.

Nettle & Bone was another great fantasy read like Paladin of Souls. Despite being straightforward and short I greatly enjoyed the story and it made me think of something like "fantasy fairy tales" as a separate subgenre of interest. In a similar vein, I enjoyed Uprooted as a recommended read after I finished the full list of was waiting for the 2025 nominees to be announced.

Pipenv to UV

2025-04-12T10:00:00-07:00

I have been (mostly) happily using Pipenv for many years now on all of my Python projects. Recently, I heard about UV so I'm giving it a try with the very small set of Python dependencies used for this blog.

Installation

I'm also moving to using a devcontainer for this blog's dependencies so for installation I'm just using the devcontainer UV feature. Then I install the Python version I want to use (3.12 here):

uv python install 3.12
Installed Python 3.12.10 in 5.25s
 + cpython-3.12.10-linux-x86_64-gnu

And initial my project:

uv init
Initialized project `chris-wells.net`

This adds a .python-version file to the repo to specify the default Python version, a main.py (which is this case I delete), a pyproject.toml file with project information, and an empty README.md.

Then, I instally my short list of requirements currently in the Pipfile.

$ uv add pelican markdown pelican-youtube
Using CPython 3.12.10 interpreter at: /usr/local/bin/python3.12
Creating virtual environment at: .venv
Resolved 24 packages in 564ms
Prepared 22 packages in 4.34s
░░░░░░░░░░░░░░░░░░░░ [0/22] Installing wheels...

warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. If the cache and target directories are on different filesystems, hardlinking may not be supported. If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.

Installed 22 packages in 72ms
+ anyio==4.9.0
+ blinker==1.9.0
+ docutils==0.21.2
+ feedgenerator==2.1.0
+ idna==3.10
+ jinja2==3.1.6
+ markdown==3.8
+ markdown-it-py==3.0.0
+ markupsafe==3.0.2
+ mdurl==0.1.2
+ ordered-set==4.1.0
+ pelican==4.11.0
+ pelican-youtube==0.3.0
+ pygments==2.18.0
+ python-dateutil==2.9.0.post0
+ pytz==2025.2
+ rich==14.0.0
+ six==1.17.0
+ sniffio==1.3.1
+ typing-extensions==4.13.2
+ unidecode==1.3.8
+ watchfiles==1.0.5

This command installs the requirements and their dependencies, updates the pyproject.toml file, and adds a uv.lock file to lock the specific dependency versions.

Since speed is a big focus of UV the warning message interested me. This is a teeny project and the performance of UV isn't relevant, but I figured it was worth learning about. After following the documentation at Using uv in Docker I ended up adding some configuration to my DevContainer Dockerfile:

# Install UV
# See: https://docs.astral.sh/uv/guides/integration/docker/#installing-uv
# See: https://docs.astral.sh/uv/guides/integration/docker/#caching
ADD https://astral.sh/uv/install.sh /uv-installer.sh
ENV UV_INSTALL_DIR="/home/$USERNAME/.local/bin/"
RUN sh /uv-installer.sh && rm /uv-installer.sh
ENV PATH="/home/$USERNAME/.local/bin/:$PATH"
ENV UV_LINK_MODE=copy
RUN --mount=type=cache,target="/home/$USERNAME/.cache/uv"

This change tells UV to use the less efficient "copy" link mode but it does so on a Docker cache volume to improve speed.

Running Code

Now that UV is installed and configured I'll try running something:

uv run pelican
[16:40:54] ERROR    Cannot load plugin `pelican_youtube`
                    Cannot import plugin `pelican_youtube`
[16:40:55] ERROR    Could not process articles/2017/aoc-2017/05-days-16-20.rst
                    /workspaces/chrxs.net/content/articles/2017/aoc-2017/05-days-16-20.rst:159: (ERROR/3) Unknown directive type "youtube"

                    .. youtube:: _cZC67wXUTs

WARNING  Unable to find '05-days-16-20.rst#day-16-one-billion-permutations-in-0-535-seconds', skipping url replacement.
WARNING  Unable to find '05-days-16-20.rst#day-17-collections-deque', skipping url replacement.
WARNING  Unable to find '05-days-16-20.rst#day-18-negative-numbers-and-str-isdigit', skipping url replacement.
WARNING  Unable to find '05-days-16-20.rst#day-19-the-internet-is-not-a-big-truck', skipping url replacement.
WARNING  Other resources were not found and their urls not replaced

Done: Processed 19 articles, 1 draft, 0 hidden articles, 2 pages, 0 hidden pages and 0 draft pages in 1.11 seconds.

Mostly good! It looks like in the 2+ years since I have updated this blog and used Pelican the pelican_youtube plugin is not working.

This turns out to be because Pelican 4.5 introduced support for plugin autodiscovery so I solved this by removing the PLUGIN_PATHS and PLUGINS settings from my pelicanconf.py.

With that corrected I'm now able to get a happy build with the latest version of Pelican!

uv run pelican
Done: Processed 20 articles, 1 draft, 0 hidden articles, 2 pages, 0 hidden pages and 0 draft pages in 1.07 seconds.

And I can run the listen server also via UV:

uv run pelican --autoreload --listen
--- AutoReload Mode: Monitoring `content`, `theme` and `settings` for changes. ---
Serving site at: http://127.0.0.1:8000 - Tap CTRL-C to stop
Done: Processed 20 articles, 1 draft, 0 hidden articles, 2 pages, 0 hidden pages and 0 draft pages in 0.99 seconds.

Ubuntu on a Macbook Pro Mid 2019 (MacBookPro15,1)

2022-02-15T20:50:00-07:00

Apple's T2 Security Chip gets itself in the way as much as possible when attempting to install Linux on a Mid 2019 Macbook Pro and a lot of the available documentation for Linux-on-the-mac online is pre-T2. After quite a bit of back and forth and eventually finding the right resources I managed to get Ubuntu installed on my MacBook Pro with everything working... except the microphone.

Initial Success and Failure

Getting started with this process was a bit of a minefield. The first resource I came across was How to Install and Dual-Boot Linux and macOS, an extremely detailed and useful documentation piece with one minor caveat (that I didn't take heed of initially):

As of right now, you cannot install Linux on the internal SSD of a newer MacBook Pro or Mac Pro (2018 or later). You can still install on an external drive, however.

Despite the warning I did manage to eventually cobble together a working base Ubuntu install using Ubuntu 20.04, a bootable USB-C drive (notably: not a USB-A adapter), and rEFInd. In the end I had a functional Ubuntu installation on the same internal SSD as my macOS install but none of the peripherals (keyboard, trackpad, speakers, touchbar, wireless, etc.) worked and Ubuntu was unable to find drivers for them. This failure/success finally led me down the proper path of seeking out a T2-supported version of Linux.

Getting it Done Right (on a T2 MacBook)

This process is mostly a mashing of various guides and notes about specific conditions for the MacBook Pro Mid 2019 (MacBookPro15,1). I've cited sources throughout the text am mostly just interested in organizing this proces all in one place.

1. Create Bootable Media

A USB-C device may be necessary for this process. Using a regular USB-A device through an adapter might not be supported for bootable media.

Download the zip files from the latest release of mbp-ubuntu (the livecd.zip file and any other livecd.zX files).
Join the zip files together: cat livecd.z* > livecd_full.zip.
And unzip them: unzip livecd_full.zip.
Create a bootable device from the resulting ISO image (e.g., with a program like Etcher).

2. Prepare Your Hard Drive

Open the Disk Utility application (located in /Applications/Utilities).
Select the internal Apple SSD entry in the left pane.
Click the Partition button.
Create a new partition (not volume) of at least 32GB with "MS-DOS (Fat)" as the format. This is where Ubuntu will be installed so choose a size appropriate for what you want to do with the system. See Partition a physical disk in Disk Utility on Mac for additional guidance on this process.

If the option to add a partition is not available it may be because additional disk space is being used by snapshots. Follow the steps below to free up this space and try to create the new partition again.

Open the Disk Utility application (located in /Applications/Utilities).
Select the primary volume group in the left pane (probably named "Macintosh HD").
Select View > Show APFS Snapshots from the menu bar.
Select all the snapshots and click the delete icon (-).
Return to the partition options (see above) and try again.

3. Boot the Bootable Media

Boot in to macOS Recovery mode and open the Startup Security Utility (see About Startup Security Utility on a Mac with the Apple T2 Security Chip).
Select "No Security" in Secure Boot.
Select "Allow booting from external and removable media" in Allowed Boot Modes.
Close the Startup Security Utility.
Insert the bootable media created previously directly in to a USB-C port.
Restart in to the Startup Disk Manager (see Change your Mac startup disk).
Select and continue from the bootable media labeled "EFI Boot". There may be more than one option with the same label -- try the last first and if it does not work just repeat the previous step and try another one.
When prompted select the Ubuntu Live environment and use the kernel with suffix "hwe-bigsur".

4. Install Ubuntu!

In the Ubuntu Live environment click the "Install Ubuntu" application on the desktop.
Set install options as desired.
On the "Installation Type" step choose "Something else" and click Continue.
Select the partition created previously (the current type should be "fat32" and the device should be something like /dev/nvme0n1p3). Be very careful to select the correct partition here as the next steps will erase it for the Ubuntu install.
Click delete partition (-).
Select the "free space" entry created by the partition removal.
Click add partition (+), set the size to 1024MB, the type to "ext4" and the mount point to /boot.
Select the "free space" entry again.
Click add partition (+), set the size to all the remaining space, the type to "ext4" and the mount point to root (/).
Select the /boot partition for "Device for bootloader installation".
Continue the installation, shutting down the computer and removing the bootable media after completion.
Start in to the Startup Disk Manager (see Change your Mac startup disk) again.
Select the new EFI Boot option.

You should now be booted in to Ubuntu with a mostly functional system!

5. Getting Wifi Working

Follow the Pre installation steps and Installation > The easy way instructions from the T2 Ubuntu Kernel repo.
Follow the Setting up steps from the t2linux wiki.

6. Celebrate!

That's it! Everything (except the microphone...) should now be working in Ubuntu -- even the touchbar! The t2linux wiki also has an audio guide with some options that may get the microphone working but I was not able to get it functional from information in that guide (or anywhere else). I'll update this space if I ever do have success!

Pretty much all of the information compiled for this post stems in one way or another from the wonderful marcosfad/mbp-ubuntu repo. Pop in there and contribute to the maintainers of that repo if this guide was helpful in any way.

Back to PHP

2018-08-11T15:00:00-07:00

A few months back I started a new job with a much greater focus on development work over IT systems management. Unfortunately this has led to a pretty big drop off in amount of time spent on personal side projects, but happily my new employer is fully supportive of open source and I am able to release much of what I work on to the wider community.

I have recently pushed three projects out:

Views Contextual Range Validator: A very simple Drupal 8 module adding a Views contextual filter for range validation, descriptively named.
PBS Passport: An OAuth2-based authorization system for Drupal 7 and PBS.org accounts.
Lightbox Campaigns: A Drupal 7 and 8 module for running custom, targeted "campaigns" using full screen "lightbox" displays.

It has been interesting returning to the world of PHP after focusing so heavily on free time personal projects in Python. I am saddened to find myself away from the (much) less verbose syntax of Python, but happy to be back in the language that I grew up hacking on.

This has also given me the opportunity to work more with custom module development in Drupal 8 and I'm excited to do more of that work in the future as the community behind Drupal has really improved the APIs. The Lightbox Campaigns module was a particularly interesting adventure because I initially developed in Drupal 8 and it seemed almost too easy to get the initial groundwork in place. Porting to Drupal 7 proved much more difficult because the same functionality had to rely on multiple other dependencies and hooks that seemed determine to enforce a particular hierarchy for entity management.

There are more projects in the pipe and planned down the road. I will continue to add things to my Projects page and Drupal stuff can always be found on my Drupal.org profile.

Pip, Pis, Pandas and Wheels

2018-05-07T20:30:00-07:00

A user attempting to install Baby Buddy submitted an interesting issue with the following error during the pipenv install process:

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    docopt==0.6.2 from https://www.piwheels.org/simple/docopt/docopt-0.6.2-py2.py3-none-any.whl#sha256=0340515c74203895f92f87702896e45424bf51dc71bf15b4748450f50be04346 (from -r /tmp/pipenv-vf5_eub9-requirements/pipenv-k7_dvsro-requirement.txt (line 1)):
        Expected sha256 49b3a825280bd66b3aa83585ef59c4a8c82f2c8a522dbe754a8bc8d08c85c491
             Got        0340515c74203895f92f87702896e45424bf51dc71bf15b4748450f50be04346

Hash checking and Pipfile.lock are a part of the pipenv toolchain and meant to verify the integrity of packages being installed. Committing the lock file is recommended practice and generally something I have not had many problems with. There are some old tickets on GitHub reporting issues with this hashing between operating systems, but the latest versions of pipenv supposedly do not have these problems.

Why is this user getting a hash match error? I had a Pi lying around, so I decided to try replicating the issue. Many hours later, I got Baby Buddy up and running on my (second) Pi and learned a lot about the Python packaging process and how it can go wrong on ARM devices.

Kernel Panics and Segfaults

In my initial tests, I couldn't even seem to get my Pi to install pipenv itself, despite the report from the user of issues after that point. The test Pi's syslog filled with errors similar to:

May  5 23:05:31 raspberrypi kernel: [18801.650989] [<8010ffd8>] (unwind_backtrace) from [<8010c240>] (show_stack+0x20/0x24)
May  5 23:05:31 raspberrypi kernel: [18801.653797] [<8010c240>] (show_stack) from [<807840a4>] (dump_stack+0xd4/0x118)
May  5 23:05:31 raspberrypi kernel: [18801.655292] [<807840a4>] (dump_stack) from [<80254a20>] (print_bad_pte+0x150/0x1b4)
May  5 23:05:31 raspberrypi kernel: [18801.658037] [<80254a20>] (print_bad_pte) from [<80256d6c>] (unmap_page_range+0x57c/0x668)
May  5 23:05:31 raspberrypi kernel: [18801.660979] [<80256d6c>] (unmap_page_range) from [<80256ea4>] (unmap_single_vma+0x4c/0x54)
May  5 23:05:31 raspberrypi kernel: [18801.664093] [<80256ea4>] (unmap_single_vma) from [<80257150>] (unmap_vmas+0x64/0x78)
May  5 23:05:31 raspberrypi kernel: [18801.667274] [<80257150>] (unmap_vmas) from [<8025d468>] (exit_mmap+0xac/0x154)
May  5 23:05:31 raspberrypi kernel: [18801.668941] [<8025d468>] (exit_mmap) from [<8011ac74>] (mmput+0x58/0x108)
May  5 23:05:31 raspberrypi kernel: [18801.670592] [<8011ac74>] (mmput) from [<80121d84>] (do_exit+0x35c/0xb9c)
May  5 23:05:31 raspberrypi kernel: [18801.672289] [<80121d84>] (do_exit) from [<8012265c>] (do_group_exit+0x4c/0xe4)
May  5 23:05:31 raspberrypi kernel: [18801.673960] [<8012265c>] (do_group_exit) from [<8012da40>] (get_signal+0x36c/0x6bc)
May  5 23:05:31 raspberrypi kernel: [18801.677190] [<8012da40>] (get_signal) from [<8010b2f4>] (do_signal+0xc4/0x3e4)
May  5 23:05:31 raspberrypi kernel: [18801.678882] [<8010b2f4>] (do_signal) from [<8010b7fc>] (do_work_pending+0xb8/0xd0)
May  5 23:05:31 raspberrypi kernel: [18801.682219] [<8010b7fc>] (do_work_pending) from [<80108094>] (slow_work_pending+0xc/0x20)
May  5 23:05:31 raspberrypi kernel: [18801.685688] BUG: Bad page map in process pip  pte:1739f75f pmd:31beb835
May  5 23:05:31 raspberrypi kernel: [18801.687473] page:ba6e525c count:1 mapcount:-1 mapping:b1ab06c9 index:0x3f1
May  5 23:05:31 raspberrypi kernel: [18801.689239] flags: 0x68(uptodate|lru|active)
May  5 23:05:31 raspberrypi kernel: [18801.690919] raw: 00000068 b1ab06c9 000003f1 fffffffe 00000001 ba6e5228 ba7194c4 00000000
May  5 23:05:31 raspberrypi kernel: [18801.694201] raw: 00000000
May  5 23:05:31 raspberrypi kernel: [18801.695918] page dumped because: bad pte
May  5 23:05:31 raspberrypi kernel: [18801.697552] addr:003f1000 vm_flags:00100073 anon_vma:b1ab06c8 mapping:  (null) index:3f1
May  5 23:05:31 raspberrypi kernel: [18801.700631] file:  (null) fault:  (null) mmap:  (null) readpage:  (null)
May  5 23:05:31 raspberrypi kernel: [18801.702198] CPU: 0 PID: 4148 Comm: pip Tainted: G    B D  C      4.14.34-v7+ #1110
May  5 23:05:31 raspberrypi kernel: [18801.705262] Hardware name: BCM2835

and

May 5 23:05:30 raspberrypi kernel: [18800.789345] Internal error: Oops: 5 [#1] SMP ARM

After trying some random things, I ran filesystem and memory tests. Everything checked out, but the Pi was consistently failing under load. I also eventually did some testing on the power supply and other aspects of the Pi, but that will have to wait for another post. I had a second Pi lying around, so I grabbed that and managed to reproduce an error similar to the reported one.

Hash Match Failures

Similar to the initial user report, the pipenv install command failed with a hash match issue. But unlike the other affected user, my failing has was for a different package.

I eventually stumbled on this very interesting blog post from Raspberry Pi Community Manager Ben Nuttall: building a faster Python package repository for Raspberry Pi users. In a nutshell, Ben explains that PyPI uses "wheels" to prebuild Python packages for installations on specific architectures, but ARM (Raspberry Pi's architecture) is not well represented by maintainers who build these wheels. Ben improved this situation greatly by developing a service to build and distribute ARM wheels: piwheels. The piwheels service is now a part of the default Raspbian distribution, making it much faster to pip install [...] Python packages out of the box!

What does this have to do hash match failures? Pipenv uses a special file, Pipfile.lock to store hash information for all dependencies in a project and committing this file is generally recommended as a security measure. The problem is that the hashes are generated from a specific source (most often PyPI) and for the large majority of instances the same source will likely be used by other users installing a project. Unfortunately -

A package from PyPI, locked on an AMD64 system will have a different hash than one from piwheels, locked on an ARM system like the Pi.

I develop Baby Buddy on my AMD64 desktop, so the lock file included in the repository has AMD64 hashes generated from PyPI. Using this same lock file on a Raspberry Pi with Rasbian results in the THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock! error from pipenv.

How can this be resolved? There are only really two options I could come up with:

Execute pipenv lock before pipenv install on ARM devices.
Include the --skip-lock flag when the pipenv install command.

Neither option is particularly ideal, as they both compromise the security aspect of the lock file (and locking can take a long time on a Pi). But for the purposes of further troubleshooting, I chose to use --skip-lock to get past this issue. And it worked!

Pip and Pandas

Happy to have sorted out the lock issue, I executed pipenv install --three --dev --skip-lock and waited for pipenv to do its thing. Sadly, the install eventually failed with the error:

Double requirement given: numpy==1.12.1 [...] (already in numpy==1.9.3 [...], name='numpy')

Numpy is a requirement of Pandas, which Baby Buddy uses for one of its graphs. I have actually thought about refactoring it out in the past as they are both pretty heavy requirements and hardly used, but I just haven't found the time.

Anyway, Baby Buddy pins Pandas below version 0.22.0 because a while back I had issue with the latest version of Pandas failing to build for Python 3.4. As it turns out, there has been a fairly recently issue with new versions of pip (10.x) and Pandas: Pandas installation problems with pip version 10. Two workaround options are provided:

use an older version of pip (9.x), or
use pip install pandas --no-build-isolation.

Before attempting these workarounds, I tried a new install of Pandas without the version pinning. Surprisingly, this worked -- Pandas and all of it's requirements installed from wheels on piwheels.

For Baby Buddy, perhaps the logical thing to do to workaround this issue will be to drop support for Python 3.4. This would allow me to remove the Pandas pinning and not have to provide special instructions for ARM-based installations. Before I could think too deeply about that, I had another error on my hands...

Psycopg2 and Pyscopg2-binary

Psycopg2 is a Python package that provides a PostgreSQL adapter. It is an important option to provide to potential Baby Buddy users and also required for the project's Heroku deployment method. Building psycopg2 requires some external development libraries and other tools that are not always present in deployment environments. In order to avoid extra dependencies, Baby Buddy uses the psycopg2-binary package. The reasoning for the two packages in explained in the Psycopg 2.7.4 release post.

Piwheels does have a psycopg2-binary wheel page, but it currently does not list any wheels. Unable to find any wheels, pip attempts to download and build the package from source, which fails with the error:

    Error: pg_config executable not found.

pg_config is required to build psycopg2 from source.  Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:

    python setup.py build_ext --pg-config /path/to/pg_config build ...

or with the pg_config option in 'setup.cfg'.

If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.

For further information please check the 'doc/src/install.rst' file (also at
<http://initd.org/psycopg/docs/install.html>).

Piwheels does provide wheels for psycopg2, but not for the latest version (2.7.4 as of this writing) so pipenv install psycopg2 fails with the exact same error message. Ultimately, this issue can be overcome by pinning psycopg2 to <2.4.7, allowing pip to find and (quickly!) install the wheel from piwheels. Not a long term solution, but it moves things forward a bit here.

ImportError: No module named 'Image'

After working around the issue with Pandas and Psycopg2, the pipenv install process works. However, further down the line in the manual install process, the gulp collectstatic command spits out this error:

Traceback (most recent call last):
  File "/home/pi/.local/share/virtualenvs/public-BuvyXxxq/lib/python3.5/site-packages/easy_thumbnails/utils.py", line 11, in <module>
    from PIL import Image
  File "/home/pi/.local/share/virtualenvs/public-BuvyXxxq/lib/python3.5/site-packages/PIL/Image.py", line 60, in <module>
    from . import _imaging as core
ImportError: libopenjp2.so.7: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  [...]
  File "/home/pi/.local/share/virtualenvs/public-BuvyXxxq/lib/python3.5/site-packages/easy_thumbnails/utils.py", line 13, in <module>
    import Image
ImportError: No module named 'Image'

Blargh. Luckily, this message is pretty verbose and has an easy fix. The libopenjp2.so.7 library is not included in Raspbian and (apparently?) included in other distributions, as I have never seen this error before. This is overcome with one additional command during the install process:

sudo apt-get install libopenjp2-7-dev

After installing this library, everything else goes smoothly and Baby Buddy runs on the Pi! Hooray!

Conclusions

Packaging is hard. I am often concerned about the fragility of the ecosystem around web development (or my little corner of it, at least). Baby Buddy is a fairly simple web app - it uses Django on the backend and Bootstrap on the frontend. It is not an SPA. It is not using React or Vue or any other complex libraries. But it still relies on a lot of external moving parts and a somewhat complicated build process. Heroku and Docker help to ease this pain, but also add time and complexities of their own.

But I still love it. And I love the community that surrounds it. It never occurred to me to run Baby Buddy on a Pi. But one user tried it and the collective knowledge of other developers, users and community members (of various communities!) helped us sort it out (temporarily, anyway).

I'm not quite sure yet how I will address this use case for Baby Buddy (suggestions welcome), but I had a damn fun time troubleshooting it! Next up, what the hell is wrong with the first Pi I tested this on...

Advent of Code 2017: Days 16 - 20

2017-12-20T10:00:00-04:00

This post is part of the series, Advent of Code 2017, where I work my way through (part of) the 2017 Advent of Code challenges in order to re-learn some of the basics of Python and reflect on some simple concepts and functionality. All my challenge code is available in cdubz/advent-of-code-2017 on GitHub.

Day 16: One Billion Permutations in 0.535 Seconds

Challenge: Day 16: Permutation Promenade
Solution: advent-of-code-2017/16

Ok, not really (: This challenge involved making specific modifications to a list 1,000,000,000 (one billion) times. Out of curiosity, the first thing I did was set up a progress bar loop to run the actual modifications all one billion times. The resulting progress bar estimated that the entire operation would take around 133 days. So... a different approach:

positions = [...]
movements = [...]
possibilities = [positions]
while True:
    positions = move(positions, movements)
    if positions in possibilities:
        break
    possibilities.append(positions)

Because the movements are always the same, eventually the positions list elements will return to their initial state. In this case, that happened on the 41st movement, making for a possibilities list of size 42. Armed with the answer to the ultimate question of life, the universe, and everything, it was much easier to determine the value of positions at the one billionth permutation:

pos = 1000000000
answer_key = pos - int(pos/len(possibilities)) * len(possibilities)
print(possibilities[answer_key])  # 1,000,000,000!

This all takes a little over half a second, as opposed to the 133 days of processing that would be needed to run all one billion permutations. Phew!

Day 17: collections.deque()

Challenge: Day 17: Spinlock
Solution: advent-of-code-2017/17

Speed is the key once again with this challenge. After optimizing my initial solution as best I could, my program was still facing an 18 hour execution time to find the correct answer. Eventually I found my way to collections.deque to speed things up tremendously. Python's documentation sums up the difference between a deque and list like so:

Deques are a generalization of stacks and queues (the name is pronounced "deck" and is short for "double-ended queue"). Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction.

Though list objects support similar operations, they are optimized for fast fixed-length operations and incur O(n) memory movement costs for pop(0) and insert(0, v) operations which change both the size and position of the underlying data representation.

Simply changing the list, which goes through 50,000,000 (50 million) insert operations, to a deque cut the execution time down a bit, but the big time saver ended up being deque.rotate. Using rotate allowed the insert operations to be replaced with much fast append operations. These two changes cut the execution time to about three minutes (from 18 hours).

Day 18: Negative Numbers and str.isdigit()

Challenge: Day 18: Duet
Solution: advent-of-code-2017/18

I was stuck with a bug while working on a piece of this challenge that had to identify the difference between an alpha and numeric string. Checking for alpha with str.isalpha() worked well enough but for some reason I wasn't having the same luck with str.isdigit():

>>> 'A'.isalpha()
True
>>> 'b'.isalpha()
True
>>> '1'.isdigit()
True
>>> '0'.isdigit()
True
>>> '-1'.isdigit()
False  # Oh no!

What these examples illustrate is that str.isdigit() does not return True for negative numbers. The Python documentation explains why (emphasis mine):

Return true if all characters in the string are digits [...]

The result for negative number is False because the negative sign itself (-) is not a digit.

Day 19: The Internet is Not a Big Truck

Challenge: Day 19: A Series of Tubes
Solution: advent-of-code-2017/19

This challenge didn't uncover any major new Python discoveries, but it was an amusing reminder of the infamous words of Ted Stevens:

I just the other day got, an internet was sent by my staff at 10 o'clock in the morning on Friday and I just got it yesterday. Why?

Because it got tangled up with all these things going on the internet commercially...

They want to deliver vast amounts of information over the internet. And again, the internet is not something you just dump something on. It's not a truck.

It's a series of tubes.

And if you don't understand those tubes can be filled and if they are filled, when you put your message in, it gets in line and its going to be delayed by anyone that puts into that tube enormous amounts of material, enormous amounts of material.

Ouch.

Day 20: Reading from a File

Challenge: Day 20: Particle Swarm
Solution: advent-of-code-2017/20

This solution makes use of earlier lessons about Manhattan Distance and the key argument for min() and max(), but beyond that nothing much new was introduced. One thing that has been important throughout this experience is reading content from a file, as most challenges include some long form of input data.

Compound statements such as if and for are as popular and well known in Python as probably any other language, but the other statement that has come in handy for almost every challenge so far is the with statement. When used with a context manager such as open(), the with statement makes it simple to do things like handle files without worrying about things like closing the file manually. The simplest example is to process each line of a file like so:

with open('file.txt') as f:
    for line in f:
        print(line)

This code opens a file.txt and assigns it to the variable f, uses a for loop to print each line of the file, and finally closes the file (automatically!). All kinds of things can be done with each line or the file itself here.

Jeff Knupp provides a very insightful read on context managers in general: Python with Context Managers.

If all lines (or other elements) of a file need to be evaluated together, the open function can be used to cram things in to a list, dictionary, or other structure easily:

lines = [line.rstrip() for line in open('input.txt').readlines()]

This simple statement will read each line of input.txt in to a list, applying rstrip each time to remove trailing data (i.e. the line breaks). Comprehensions like this can quickly accomplish basic data manipulation to turn file lines in to more easily manageable structures.

Advent of Code 2017: Days 11 - 15

2017-12-15T08:00:00-04:00

This post is part of the series, Advent of Code 2017, where I work my way through (part of) the 2017 Advent of Code challenges in order to re-learn some of the basics of Python and reflect on some simple concepts and functionality. All my challenge code is available in cdubz/advent-of-code-2017 on GitHub.

Day 11: Navigating a Hexagon Grid

Challenge: Day 11: Hex Ed
Solution: advent-of-code-2017/11

I fought with how to create and navigate a hexagon grid for a bit before I stumbled on an extremely useful post by Chris Schetter: Hexagon grids: coordinate systems and distance calculations. Working in 2d grids (with four possible movement directions) in Python is fairly easy with some dictionaries, but a hexagon grid means there are six potential directions and simple calculations of x,y coordinates won't quite get the job done. What Chris explains so well is a simple concept: flip the grid and add a z-axis.

For my solution, I "flip" the grid by turning it to the right twice such that north and south become east and west:

  \ n  /
nw +--+ ne              sw +--+ nw
  /    \                \ /    \ /
-+      +-    ---->    s +      + n
  \    /                / \    / \
sw +--+ se              se +--+ ne
  / s  \

This puts north and south along the x-axis instead of the y-axis. Next, adding the z-axis and shifting all axises 120 degrees means that each hex can be made up of three points instead of two. Again, I refer to Chris's post and this image that illustrates the concept wonderfully:

Credit Chris Schetter (Hexagon grids: coordinate systems and distance calculations)

Armed with three coordinates, it is now much easier to navigate the hex grid and calculate the difference between two locations with Python. First, two of the three coordinates need to be modified for each type of movement:

def move(x, y, z, direction):
    if direction == 'n':
        x += 1
        z -= 1
    elif direction == 'ne':
        x += 1
        y -= 1
    elif direction == 'se':
        y -= 1
        z += 1
    elif direction == 's':
        x -= 1
        z += 1
    elif direction == 'sw':
        x -= 1
        y += 1
    elif direction == 'nw':
        y += 1
        z -= 1
    return x, y, z

This function takes the current position (x, y, z) and returns modified coordinates based on the direction being moved. After all movement is a complete, a simple formula can be used to calculate the shortest number of movements between two points on the hex grid.

max(from_x - to_x, from_y - to_y, from_z - to_z)

That's all there is to it!

Day 12: Recursion

Challenge: Day 12: Digital Plumber
Solution: advent-of-code-2017/12

When faced with a recursion problem, I always tend to struggle a bit to build a mental model of what my code needs to do. This challenge provided a number of "programs" (identified by numbers) interconnected by "pipes" that could be used to connect programs at any depth. Recursion is therefore necessary to figure out all available connections.

I achieved this with a function that takes all the pipes, a "parent" program (number) to evaluate ("plumb"), a list to record connections, and a list to record already plumbed programs:

def plumb(pipes, parent, connections, plumbed):
    connections.append(parent)
    plumbed.append(parent)
    for child in pipes[parent]:
        if child not in plumbed:
            connections.append(child)
            plumbed.append(child)
            plumb(pipes, child, connections, plumbed)
    return connections

The recursion is triggered on line 7, but only if the program being evaluated (child) has not already been plumbed. This is tracked by passing the connections and plumbed lists in to each new execution of plumb in order to maintain an ongoing list and prevent infinite recursion.

While this approach worked, I am inclined to think there are probably better ways to deal with recursion problems like this in Python - a challenge for another day.

Day 13: Shallow vs. Deep Copy

Challenge: Day 13: Packet Scanners
Solution: advent-of-code-2017/13

Warning: my solution to this challenge is horribly inefficient. I hope to return to this challenge some day when I have a little more time (I'm looking after my infant son right now so time is pretty tight) to figure out a more efficient way to do it. My solution took an hour and four minutes to complete. By contrast, the fastest finisher for the day completed the challenge (including reading the problem and writing the code) in about five minutes...

The most interesting discovery I made while working on this was about the difference between a "deep" and "shallow" copy in Python. Python, unlike other languages such as PHP (where most of my experience is), uses "names" instead of "variables" when assigning values. I already had a vague understanding of this, but this challenge quickly revealed just how vague. The concept is thoroughly explained by Mathieu Larose's post: Understanding Python Variables.

This was as issue when I was trying to modify a copy of a dictionary so I could reuse the original later in the code. Here is a basic example of what happens in Python:

>>> a = {1:2, 3:4, 5:6}
>>> b = a
>>> print(b)
{1: 2, 3: 4, 5: 6}
>>> a[7] = 8
>>> print(b)
{1: 2, 3: 4, 5: 6, 7: 8}

In assigning b = a, Python actually links names a and b to the same dictionary ({1:2, 3:4, 5:6}, initially). When I ran in to this problem, my approach was to copy the dictionary using dict.copy(). This process, called a shallow copy, still actually creates references for each item in the dictionary, so changes to existing dictionary items continue to happen in both places:

>>> a = {1:2, 3:4, 5:6}
>>> b = a.copy()
>>> print(b)
{1: 2, 3: 4, 5: 6}
>>> a[1] = 0
>>> print(b)
{1: 0, 3: 4, 5: 6}

The alternative is a deep copy, which copies the dictionary and all of it's items to a new dictionary:

>>> a = {1:2, 3:4, 5:6}
>>> b = deepcopy(a)
>>> print(b)
{1: 2, 3: 4, 5: 6}
>>> a[1] = 0
>>> print(b)
{1: 2, 3: 4, 5: 6}
>>> print(a)
{1: 0, 3: 4, 5: 6}

There we go! Python's "copy" documentation explains this all in very clear terms and ultimately it seems to be something that a developer like me, coming from PHP, needs to learn and relearn a few times.

Day 14: Hexadecimal to Binary

Challenge: Day 14: Disk Defragmentation
Solution: advent-of-code-2017/14

A key aspect of this challenge is to convert a group of hexadecimal digits to their binary representations with zero-prefixed padding. A simple way to do this in Python is as follows:

binary = [bin(int(digit, 16))[2:].zfill(4) for digit in hex_digits]

The [x for x in y] structure should look very familiar to Python users - this is a list comprehension that will perform operations on each item in a for-loop and create a list structure containing the results.

The first operation being performed in this case is an int() on each digit in hex_digits:

>>> digit = '1'
>>> int(digit, 16)
'1'
>>> digit = 'f'
>>> int(digit, 16)
'15'

The second parameter for this int() operation sets the base (or radix) for the operation. Hexadecimal is base 16, meaning it contains 16 digits: 0 - 9 and a - f, so init('f', 16) results in 15.

Next, bin() converts this result to a binary string:

>>> bin(1)
'0b1'
>>> bin(15)  # 'f'
'0b1111'

This challenge does not use the 0b prefix that bin() returns, so a slice is used to exclude the first two characters: [2:].

Lastly, all binary string representations need to be four characters so string.zfill() is used to pad the string with zeros:

>>> '1'.zfill(4)
'0001'
>>> '1111'.zfill(4)  # 'f'
'1111'

The result of all this is a list of binary representations of each hexadecimal digit:

>>> hex_digits = '218f88a3f546f710c0aabb767316aebd'
>>> binary = [bin(int(digit, 16))[2:].zfill(4) for digit in hex_digits]
>>> print(binary)
[
    '0010', '0001', '1000', '1111',
    '1000', '1000', '1010', '0011',
    '1111', '0101', '0100', '0110',
    '1111', '0111', '0001', '0000',
    '1100', '0000', '1010', '1010',
    '1011', '1011', '0111', '0110',
    '0111', '0011', '0001', '0110',
    '1010', '1110', '1011', '1101'
]

Day 15: progressbar2

Challenge: Day 15: Dueling Generators
Solution: advent-of-code-2017/15

This challenge led me to progressbar2, a simple Python utility for quickly creating text-based progress bars from standard loops. While not terribly valuable for expanding my Python skills, it is certainly a nice utility to know about. My original code included:

for i in range(0, 5000000):
    # do stuff

This executes five million loops and, with the actual code, takes a little over 30 seconds to run. As I was fighting with an odd bug in my code, it became more and more annoying to stare at a blank terminal for 30+ seconds while the code executed. During one of those 30 second waits, I went searching for how to create a progress bar in Python.

Lucky for me, it is incredibly simple! After installing progressbar2 via pip (pip install progressbar2), I only had to make a few small changes to get a result:

from progressbar import ProgressBar
bar = ProgressBar
for i in bar(range(0, 5000000)):
    # do stuff

Now, as the code executes, I see a simple text progress bar that updates with each pass of the loop and doesn't add any significant time to the execution. Here is an example of the output:

36% (1815386 of 5000000) |##############                          | Elapsed Time: 0:00:12 ETA:  0:00:21

I only wish I had made this discovery on Day 13 (my unoptimized solution took over an hour to run)...

Advent of Code 2017: Days 6 - 10

2017-12-10T12:30:00-04:00

This post is part of the series, Advent of Code 2017, where I work my way through (part of) the 2017 Advent of Code challenges in order to re-learn some of the basics of Python and reflect on some simple concepts and functionality. All my challenge code is available in cdubz/advent-of-code-2017 on GitHub.

Day 6: max(x, key=y)

Challenge: Day 6: Memory Reallocation
Solution: advent-of-code-2017/06

Another simple revelation on built-ins from this challenge: the key argument can be used to modify what is evaluated by the max function. This concept is explained in detail as part of the accepted answer to this Stack Overflow post: python max function using 'key' and lambda expression.

For Python dictionaries, this means that the get() method can be used to return the key of the maximum value in a dictionary. This works because the max function sends each of the dict keys to the dict.get() method and evaluates that result instead of the key itself. Without this argument, max will actually evaluate the dictionary's keys, which isn't terribly useful:

>>> d = {0: 1, 2: 5, 3: 4}
>>> max(d)
3  # This is the *maximum value of the dictionary's keys*.
>>> max(d, key=d.get)
2  # This is the *key of the maximum value in the dictionary*.

Day 7: RuntimeError: dictionary changed size during iteration

Challenge: Day 7: Recursion
Solution: advent-of-code-2017/07

My initial approach to this problem was to organize all "programs" in to a dictionary and whittle away at any program that wasn't a child of another program:

programs = {'<name>': {'weight': <int>, 'children': []}, ...}
for program, data in programs.items():
    for child in data.get('children'):
        if child in programs:
            del programs[child]
print(list(programs)[0])

The end result should be a single entry in programs that does not appear in the children list of any other program - aka the base of the recursion.

After running this successfully a few times and submitting my answer, I did some other things before coming back for part two. To my surprise, when I tried to run the same code again, it produced an error: RuntimeError: dictionary changed size during iteration

While trying to figure out why this was happening all of a sudden, I realized that I had, in the course of doing other things, changed to an environment with Python 3 as the default python, whereas my original environment used Python 2.

This error happens because dict.items() is a simple list in Python 2, but it is a view in Python 3. If my loop had not been modifying the contents of the dictionary, this would not have been a problem because either type can be iterated in the same way. However, since I was modifying the dictionary in place, Python 3 raises the RuntimError because the actual dictionary object (not the list copy of its items, as with Python 2) is being modified while the code is still in the loop.

The quick fix for this issue is to make a copy of the dictionary and iterate over that instead. E.g. by changing the for-loop like so:

for program, data in dict(programs).items():
    [...]

Day 8: Regular Expressions

Challenge: Day 8: I Heard You Like Registers
Solution: advent-of-code-2017/08

I have used regular expressions in the past two challenges. Every time I need a bit of regex I think to myself, "I should learn more about this in depth". I have yet to scratch that particular itch, but my general knowledge about them came in handy for this challenge.

The challenge provides line-by-line string instructions for modifying values, like this example (from the challenge):

b inc 5 if a > 1
a inc 1 if b < 5
c dec -10 if a >= 1
c inc -20 if c == 10

Looking at this, there are four important pieces of information:

The "variable" to be modified (a, b, c, etc...).
How to modify that variable ("inc" for increase, "dec" for decrease).
The amount to modify the variable by.
An expression to decide whether or not to perform the actual modification.

Regular expressions to the rescue! The first thing I tend to do when trying to write a regular expression is pull up one of the many useful websites that will evaluate expressions in real time. In this case I used regex101.com, which will take the text to be searched, highlight the relevant parts, and provide explanations as you type the expression. I ended up with the following expression for the challenge:

([a-z]+) (inc|dec) (-?\d+) if ([a-z]+) (.+) (-?\d+)

This regex, the example input, and all of the useful context can be viewed at the following URL: https://regex101.com/r/O3xLTL/2.

Using Python's re module and some small modifications to the expression, the relevant capture groups can be organized in to more descriptive variables:

import re

with open('input.txt') as f:
    for line in f:
        result = re.search('(?P<register>[a-z]+) (?P<op>inc|dec) (?P<amt>-?\d+) '
                           'if (?P<cond_register>[a-z]+) (?P<cond>.+) '
                           '(?P<cond_amt>-?\d+)', line.strip())
        [...]

Python's re.search uses the syntax (?P=<name>[expression]) to enable this grouping. result.group('register') run on the first line produces "b", result.group('cond') run on the second line produces "<", and so on. This grouping further enhances the usefulness of regular expressions and makes building expressions (and evaluating functionality in general) a lot of fun.

Day 9: Lookahead, Lookbehind

Challenge: Day 9: Stream Processing
Solution: advent-of-code-2017/09

Regular expressions to the rescue again! Part one of this challenge was fairly straightforward and easy to complete with two passes of expressions to remove characters from a long string.

Part two was a bit more difficult because it was necessary to count the removed characters. I wanted to build on part one's character removal by comparing the string length before and after instead of trying to significantly modify the expression to use (and add up) grouping. The problem with this approach is that it requires matching strings that shouldn't be included in the count of removed characters because the task is to remove characters between brackets (< and >):

>>> import re
>>> string = '<characters>'
>>> print(re.sub(r"<[^>]+>", "", string))
   # (an empty string)

How can the expression <[^>]+> be changed to capture what is inside the brackets, but not the brackets themselves? With lookaround assertions! A lookahead assertion has the format (?=...) for positive assertions and (?!...) for negative assertions and a lookbehind uses (?<=...) and (?<!...) respectively.

Replacing the surrounding brackets with a single lookbehind solves the problem of capturing the brackets:

>>> import re
>>> string = '<characters>'
>>> print(re.sub(r"(?<=<)[^>]+", "", string))
<>  # only the brackets remain

Now the expression produces the desired effect - it removes all characters inside brackets, but not brackets themselves. This works because the positive lookbehind assertion ((?<=<)) ensures that the pattern matches, but does not actually match it. The remaining portion of the expression ([^>]+) does the actual matching of any character other than > (aka, everything inside the brackets!).

Day 10: functools.reduce()

Challenge: Day 10: Knot Hash
Solution: advent-of-code-2017/10

Part two of this challenge dealt in part with performing operations on each pair of two items cumulatively in a list of items. For example, multiplying all values in the list [1, 2, 3, 4, 5] should yield 120 because 1 * 2 = 2 * 3 = 6 * 4 = 24 * 5 = 120. An easy way to do this in Python is to iterate the list and deal with each element one-by-one:

>>> l = [1, 2, 3, 4, 5]
>>> result = l[0]
>>> for number in l:
>>>     result *= number
>>> print(result)
120

While this method certainly is not complicated, the code would have to be modified to work with other iterables and in many cases may not be so clean. As an alternative, functools.reduce() can perform the same action in a single line, on any iterable:

>>> from functools import reduce
>>> from operator import mul
>>> l = [1, 2, 3, 4, 5]
>>> print(reduce(mul, l))
120

This example uses operator.mul for a simple multiplication function. But reduce() will work with any function that accepts two arguments (including lambdas!):

>>> from functools import reduce
>>> l = [1, 2, 3, 4, 5]
>>> print(reduce(lambda i, j: i * j, l))
120

Advent of Code 2017: Days 1 - 5

2017-12-05T10:00:00-04:00

This post is part of the series, Advent of Code 2017, where I work my way through (part of) the 2017 Advent of Code challenges in order to re-learn some of the basics of Python and reflect on some simple concepts and functionality. All my challenge code is available in cdubz/advent-of-code-2017 on GitHub.

Day 1: zip()

Challenge: Day 1: Inverse Captcha
Solution: advent-of-code-2017/01

This challenge taught me about Python's built-in zip function. The basic goal is to compare values in a list with specific positional relationships to other values (e.g. next to, X steps from, etc.) in the same list. zip assists with this task by combining multiple lists in to tuples. My initial solution used a construct similar to:

digits = [1, 2, 3, 4, 5]
total = 0
for a, b in zip(digits[::1], digits[1::1]):
    if a == b:
        total += a

This code uses zip and Python's list indexing to evaluate each element in the digits list against the next element in the list by creating and combining two lists: one starting from the first digit in digits and one starting from the second digital in digits. Here are what the different parts of this process look like:

>>> digits = [1, 2, 3, 4, 5]
>>> digits[::1]
[1, 2, 3, 4, 5]
>>> digits[1::1]
[2, 3, 4, 5]
>>> list(zip(digits[::1], digits[1::1]))
[(1, 2), (2, 3), (3, 4), (4, 5)]

Day 2: itertools.combinations()

Challenge: Day 2: Corruption Checksum
Solution: advent-of-code-2017/02

I have been a bit slow to pick up on Python's powerful iteration tools (itertools) so it took me a while (and a bit of searching, of course) to come up with a solution for this challenge. The basic goal is to evaluate all possible combinations of integers in a list. While I had some vague understanding of an efficient way to do this, my initial solution was extremely basic and inefficient:

digits = [2, 4, 5, 6, 9]
total = 0
for i in digits:
    for j in digits:
        if i != j and i % j == 0:
            total += i/j

This code has two major issues:

1. It compares every number against every other number, regardless of whether or not those numbers have already been compared. In this case, 2 is compared to 4 and 4 is compared to 2 - two comparisons where only one is needed. This issue will compound and cause greater inefficiencies with a larger list.

1. It assumes that no digit will be repeated. With, for example, digits = [2, 2, 3], the two initial values would never be evaluated because they are equal. While this wouldn't actually cause any problems with the original list provided, it is a pretty silly way to do things.

itertools.combinations can make this process more efficient without creating particularly complex code. This method takes a list and returns all possible unique combinations based on element position instead of value - meaning it solves both of the above issues. A refactoring of the above code using this method looks like this:

import itertools

digits = [2, 4, 5, 6, 9]
total = 0
for i, j in itertools.combinations(digits, 2):
    if i % j == 0:
        total += i/j
    if j % i == 0:
        total += j/i

Now, instead of a nested loop over the same list, there is only a single loop that evaluates list elements based on position instead of value.

This was also a valuable reminder about big O notation. The original code above is O(N^2), meaning it grows in complexity exponentially as the size of the list increases. The revised code, on the other hand, is O(N), so its complexity is only linearly related to the size of the list.

Day 3: Manhattan Distance

Challenge: Day 3: Spiral Memory
Solution: advent-of-code-2017/03

When I originally started this challenge I did not realize that Manhattan Distance was a link in the prompt. I had never heard this term before and a StackExchange question, How to calculate the Euclidean and Manhattan distance?, clued me in to the process for calculating the distance in a matrix. I thought about involving numpy to make things easier on myself, but decided to try solving it first without any external libraries.

The solution to this problem is very simple once the (x,y) coordinates of the number being evaluated are known (because it is being compared to 0,0). To do this I basically just brute-forced my way through creating a dictionary full of dictionaries, one for each column of the matrix, by checking for existing data in the N, S, E, and W directions of the "2D" array. With the matrix built, the solution is calculated by using the abs(x2 - x1) - abs(y2 - y1) formula.

Completing this challenge was mostly just a reminder that I need to dive in to numpy eventually. My solution is fairly clean, but horribly inefficient and I'm sure it could be improved upon greatly.

Day 4: set()

Challenge: Day 4: High-Entropy Passphrases
Solution: advent-of-code-2017/04

This challenge was another reminder about a simple and useful Python built-in type: set(). set() can be used to quickly determine if a list has duplicate items by comparing the length of the original list and its set because a set is a collection of distinct items:

>>> digits = [1, 1, 2, 3, 4, 5]
>>> digits
[1, 1, 2, 3, 4, 5]
>>> set(digits)
{1, 2, 3, 4, 5}
>>> len(digits) == len(set(digits))
False

Note that set is a class separate from list and dict. I originally attempted to check for equality between a list and set but this did not work, even with elements in the same order, because the two classes are not the same:

>>> list1 = [1, 2, 3, 4]
>>> list2 = [1, 2, 3, 4]
>>> list1 == list2
True
>>> list1
>>> [1, 2, 3, 4]
>>> set1 = set(list1)
>>> set1
{1, 2, 3, 4}
>>> list1 == set1
False

Day 5: Profiling

Challenge: Day 5: A Maze of Twisty Trampolines, All Alike
Solution: advent-of-code-2017/05

The challenge of jumping around a dict and updating its values was fairly simple. The execution of the script for part one took no time at all, but part two complicated the task and running the script ends up taking 15 - 20 seconds or more (on my very old laptop).

Curious to see if there was anything I could do to improve this, I first turned to Python's built-in profiler cProfile by running the suggested python -m cProfile <script> command:

python -m cProfile maze_part_two.py
26395586
     6 function calls in 18.451 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1   18.450   18.450   18.451   18.451 maze_part_two.py:11(<module>)
    1    0.001    0.001    0.001    0.001 maze_part_two.py:12(<dictcomp>)
    1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
    1    0.000    0.000    0.000    0.000 {method 'read' of 'file' objects}
    1    0.000    0.000    0.000    0.000 {method 'split' of 'str' objects}
    1    0.000    0.000    0.000    0.000 {open}

This didn't tell me much of anything because cProfile works at the function level. Since my solution is a simple script that does not use any functions, all this really says is: "the script took 18.451 seconds to run". This would definitely be a very useful tool to quickly identify problem areas in a larger code base, but is not helpful here.

To find more detail, I turned to the line_profiler module. As the name implies, line_profiler will provide line-by-line profiling for a Python script. I still had to refactor the code slightly to put it in a function that would allow me to use line_profiler's decorator:

@profile
def run():
    [...]

if __name__ == '__main__':
    run()

The @profile decorator is what tells line_profiler to take a closer look at the function. The evaluation is a two step process - the kernprof command generates an lprof file that line_profiler can then translate in to useful output:

kernprof -l maze_part_two.py
26395586
Wrote profile results to maze_part_two.py.lprof

This took about three minutes to run because of the detailed evaluation. Next, line_profiler can translate the lprof file using python's -m flag:

python -m line_profiler maze_part_two.py.lprof
Timer unit: 1e-06 s

Total time: 191.125 s
File: maze_part_two.py
Function: run at line 13

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
                                         @profile
                                         def run():
       1          886    886.0      0.0      maze = [...]
       1            2      2.0      0.0      pos = 0
       1            1      1.0      0.0      steps = 0
       1            1      1.0      0.0      while True:
26395587     24430262      0.9     12.8          try:
26395587     29073318      1.1     15.2              move = maze[pos]
       1            3      3.0      0.0          except KeyError:
       1           72     72.0      0.0              print(steps)
       1            3      3.0      0.0              break
26395586     24881530      0.9     13.0          last_pos = pos
26395586     26706364      1.0     14.0          pos = last_pos + move
26395586     26374692      1.0     13.8          if move >= 3:
13057988     16389704      1.3      8.6              maze[last_pos] -= 1
                                                 else:
13337598     15591510      1.2      8.2              maze[last_pos] += 1
26395586     27676460      1.0     14.5          steps += 1

There we go! Much more information than cProfile provided. The "Hits" column contains the number of times each line was run, "Time" the total time spent on the line (in the timer unit (1e-06s in this case)), and the remaining columns build on that data.

Advent of Code 2017

2017-11-30T19:45:00-04:00

I taught myself Python a few years ago by following the wonderful Learn Python the Hard Way (LPTHW) series by Zed A. Shaw. Since then, I have spent a decent amount of time in Python largely in the Django framework. Django is a lot of fun to work with because it abstracts away much of the complexities of developing a full-featured web application in Python. In this way, however, it has also led me to forget some of the basic Python that I learned through LPTHW.

In order to recapture some of those early lessons (and maybe learn a few more Python 3 specific ones), I worked through (part of) the 2017 Advent of Code, a 25-day, language agnostic programming challenge series developed by Eric Watsl. I originally thought about using the series to learn a new language, but eventually realized that I still have a long way to go in Python.

This series explores my (often very basic) revelations and lessons learned while completing 20 of the 25 days of challenges (vacation travel cut the series short for me). The code I used for each day is available on GitHub. Also check out the advent-of-code and advent-of-code-2017 GitHub topics to see the many solutions people have developed in various languages.

Consistent Selenium Testing in Python

2017-09-01T10:10:00-04:00

Back in April, I learned about Timestrap, a self-hostable, Django-based time-tracking project from a post on HackerNews by Isaac Bythewood. As I have been learning Python in the past year or so, I reached out to Isaac and started contributing to the project. After getting familiar with the core application, I turned my attention to testing and eventually found my way to Selenium, a collection of browser automation tools used for frontend testing.

I had never worked with Selenium or other automated testing products, so it struck me as a great opportunity to get my feet wet in something new. After getting things up and running, we quickly learned that the test results were quite inconsistent across development environments - even to a point that occasionally tests would succeed when run individually, but fail with the full test case.

After much trial and error, we have settled on a (mostly) consistent setup for testing with Selenium, Python and SauceLabs. This produces much better results than testing in development environments and crossing fingers during CI. Hopefully this primer will help others facing similar challenges (as we had a lot of trouble finding good material on the subject).

Getting Started
Key Selenium Functionality
Improving Consistency with SauceLabs
Putting It All Together

Getting Started

Use pip to install the selenium package (perhaps in a virtual environment):

pip install selenium

Selenium needs a WebDriver before it can do anything useful. There are currently drivers for Firefox, Chrome, Edge and Safari. We originally started out with Firefox's geckodriver, but in initial attempts to fight inconsistency moved to chromedriver hoping for better results. Ultimately, both seem to have their shortfalls but we have stuck with chromedriver since the original change so that is what I will use in examples here.

Installing Chromedriver

The installation process is pretty simple, chromedriver just needs to be executable on the development system so Selenium can interact with it during testing. Check the ChromeDriver Downloads page for the latest version of the driver to download and install. In Linux, this may look like this:

curl -L https://chromedriver.storage.googleapis.com/2.32/chromedriver_linux64.zip -o chromedriver.zip
sudo mkdir -p /usr/local/bin/
sudo unzip chromedriver.zip -d /usr/local/bin/
sudo chmod +x /usr/local/bin/chromedriver

The above set of commands

downloads chromedriver,
places it in a common $PATH location, and
sets it to be executable.

Driving Chrome with Python

With chromedriver ready to go, all that is left is to import the WebDriver package from Selenium and tell it to use chromedriver. E.g.

from selenium import webdriver

driver = webdriver.Chrome()

Running this code should invoke a window in Chrome, but nothing will happen because this example does not give the WebDriver any instruction.

Let's try a simple task: getting the "word of the day" from Merriam-Webster's website. A quick look at the source of M-W's word of the day page reveals where the word can be found in markup:

<article>
...
    <div class="quick-def-box">
      <div class="word-header">
        <div class="word-and-pronunciation">
          <h1>confrere</h1>
          ...
        </div>
      </div>
...
</article>

So the actual word of the day, "confrere" today, is found in a h1 child of a div element with the class word-and-pronunciation. Searching the page reveals that this class is unique, so it can be used by Selenium to identify the element and get its content like so:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://www.merriam-webster.com/word-of-the-day')
element = driver.find_element_by_css_selector('.word-and-pronunciation h1')
print(element.text)
driver.close()

Running the above should invoke a Chrome window that loads the word of the day page and then closes. The Python script should output the word before exiting. And there you have it! This example uses a CSS selector with Selenium's find_element_by_css_selector method, but there are many other find_element_by_* methods available for page "navigation".

Going Headless

For the examples ahead it will be useful to continue using chromedriver as is, however for actual testing, "headless" mode will be the way to go. At the time of this writing, headless requires Chrome beta to be installed on the development system. Let's modify the previous example a bit:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--headless')
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://www.merriam-webster.com/word-of-the-day')
element = driver.find_element_by_css_selector('.word-and-pronunciation h1')
print(element.text)
driver.close()

Running this code should print out the word of the day, just as the previous example did, but this time no Chrome window will appear on screen. This is because the new version uses the WebDriver.Chrome.options class to provide arguments to the Chrome (beta) binary using chrome_options. The only option passed here is --headless, which tells Chrome to execute the actions without rendering anything.

Comparing the speed of these two examples wouldn't reveal much, but when used in a real set of tests, with possibly hundreds of separate tests for an app, the time savings will be incredible.

Key Selenium Functionality

There are lots of important Selenium classes and methods that will be used extensively for testing web pages. Here is a short list of some key functionality to know about -

Finding Elements
Sending Input
Clearing Input
Waiting

Finding Elements

The example above uses WebDriver.find_element_by_css_selector and there are eight of these methods in total (plus eight more in plural form):

find_element_by_class_name
find_element_by_css_selector
find_element_by_id
find_element_by_link_text
find_element_by_name
find_element_by_partial_link_text
find_element_by_tag_name
find_element_by_xpath

All of these methods are pretty descriptive (and long), so a nice helper is the WebDriver.common.By class. By can replace the longer form methods with a simpler shorthand. The previous code example could be replaced with:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get('https://www.merriam-webster.com/word-of-the-day')
element = driver.find_element(By.CSS_SELECTOR, '.word-and-pronunciation h1')
print(element.text)
driver.close()

While this code is not necessarily shorter, I suggest taking it a bit further and creating a wrapper method for finding elements. This should significantly reduce the effort of typing these methods out as test size and complexity increases. Here is an example wrapper I have used in test cases:

def find(self, by, value):
    elements = self.driver.find_elements(by, value)
    if len(elements) is 1:
        return elements[0]
    else:
        return elements

This uses the plural find_elements method and returns either a list or a single item depending on what is found. With this, I can use find(By.ID, 'my-id') instead of driver.find_element_by_id('my-id'). This form should produce much cleaner code, particularly when jumping between the various available find methods.

Sending Input

Most web app projects will deal with some degree of input and Selenium can support that fairly well. Every WebElement class (the result of the various find_element methods) has a send_keys method that can be used to simulate typing in an element. Let's try to use this functionality to search "Python" on Wikipedia -

A quick look at Wikipedia's page source reveals that the search input element uses the id searchInput. With this, Selenium can find the element and send some keys to it:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get('https://www.wikipedia.org/')
el = driver.find_element(By.ID, 'searchInput')
el.send_keys('Python')

The above code should result in an open Chrome window with the Wikipedia page loaded and "Python" in the search input field. This windows stays open because the code does not include the driver.close() command that is used in previous examples.

There are a couple of different ways to actually submit a form. In general I have found no real difference between any of the options, but I tend to fall back on locating and "clicking" the form's submit button when possible. Here are some of the ways submission can be accomplished:

Submitting the form element

Taking another look at the Wikipedia source, the search form has a simple ID: search-form. This ID can be used with the WebElement.submit() method to submit the form.

Add the following to the previous example from Using Input:

form = driver.find_element(By.ID, 'search-form')
form.submit()

Running the code should leave you with a Chrome window open to Wikipedia's results page for Python.

Clicking a form submit button

The Wikiedpia search page includes a fancy, styled submit button for searching. It does not have a unique ID, so the code will need to use some other method to identify and "click" the button. It is the only button element inside search-form, so it can be easily targeted with a CSS selector.

Add the following to the example from Using Input:

button = driver.find_element(By.CSS_SELECTOR, '#search-form button')
button.click()

As above, this code should produce Wikipedia's results page for Python.

Pressing the enter key

Lastly, Selenium has a set of key codes that can be used to simulate "special" (non-alphanumeric) keys. These codes are found in WebDriver.common.keys. In order to submit the form, the code will need to use the return (or enter) key, so a revised version of the Wikipedia search code looks like this:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()
driver.get('https://www.wikipedia.org/')
el = driver.find_element(By.ID, 'searchInput')
el.send_keys('Python')
el.send_keys(Keys.RETURN)

Just like the two previous examples, this script should exit leaving a Chrome page open to the Wikipedia search results for "Python".

This is perhaps the cleanest way to get a form submitted because it doesn't require finding other elements, but a thorough tester may want to consider testing multiple submission methods to ensure functionality.

Clearing Input

While Selenium does offer a WebElement.clear() method, its implementation is inconsistent across browsers and its behavior can be defined differently depending on the app and element being tested. For these reasons, I don't think it should be used to clear form input fields. Instead, Selenium's Keys class can be used to simulate pressing the backspace key multiple times in a field.

Here is a simple function to handle this -

from selenium.webdriver.common.keys import Keys


def clear(element):
    value = element.get_attribute('value')
    if len(value) > 0:
        for char in value:
            element.send_keys(Keys.BACK_SPACE)

This clear function will take a WebElement, get the length of its value attribute, and simulate hitting the BACK_SPACE until all text is removed from the field.

Let's use Selenium to load Google and search for "selenium". Google's search input element does not have a unique ID or class, but it does use a name attribute with the value "q". This can be used to find the element and send the keys. Expanding on the previous example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys


def clear(element):
    value = element.get_attribute('value')
    if len(value) > 0:
        for char in value:
            element.send_keys(Keys.BACK_SPACE)

driver = webdriver.Chrome()
driver.get('https://www.google.com/')
el = driver.find_element(By.NAME, 'q')
el.send_keys('selenium')
el.send_keys(Keys.RETURN)

This should produce the Google search results page for "selenium".

On the results page, the search field still has a name value of "q" and now is pre-filled with "selenium" for a value. Although the name has not changed, Selenium will need to find the element again because the page has changed. Add the following to the code to locate the element and use the custom clear() function to clear it:

el = driver.find_element(By.NAME, 'q')
clear(el)

And it's gone!

Overall, this BACK_SPACE should be much more reliable than the WebElement.clear() method.

Waiting

"Waiting" in Selenium can be a deceptively complex problem. Up to this point, all examples have relied on Selenium's own ability to wait for a page to finish loading before taking any particular action. For simple tests, this may be a perfectly sufficient course. But as tests and applications become more complex this method may not always do the job.

Selenium provides some useful tools for addressing this issue -

Implicit waits

The easiest way to add some wiggle room is the WebDriver.implicitly_wait() method. This method accepts an integer input that defines how many seconds to wait when executing any of the find_element methods.

The default implicit wait is zero (or no wait), so if a particular element is not found immediately Selenium will raise a NoSuchElementException. Let's try to find an element with a name attribute "query" on GitHub (there isn't one):

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()
driver.get('https://www.github.com/')
el = driver.find_element(By.NAME, 'query')

This code should result in a NoSuchElementException pretty quickly after Chrome loads GitHub's homepage.

Now, let's try the code below, which sets an implicit wait time of five seconds for the same impossible task:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()
driver.implicitly_wait(5)
driver.get('https://www.github.com/')
el = driver.find_element(By.NAME, 'query')

This code will produce the exact same exception, but it will wait five seconds before doing so.

While these examples paint a very simple picture, the reality is that various conditions of any test environment or application will impact Selenium's ability to determine when a page is loaded or whether or not an element exists.

I recommend all tests set a 10 second implicit wait time. This should help to prevent intermittent exceptions caused by issues with underlying elements like network connection or buggy web servers.

Expected conditions (explicit waits)

When implicit waits are not enough, expected conditions are extremely valuable. The WebDriverWait class provides the until() and until_not() methods that can be used with expected_conditions to create more complex and nuanced wait conditions.

There are many expected conditions available, but the one that I have frequently come back to in my testing is presence_of_element_located().

presence_of_element_located() will take an object describing a method and locator and return true if the object exists in the DOM. This can be used with WebDriverWait.until() and a wait time (in seconds) like so:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec

driver = webdriver.Chrome()
WebDriverWait(driver, 5).until(ec.presence_of_element_located((By.ID, 'html-id')))

For a real example, the website webcountdown.net creates a countdown timer that creates a pop-up in the DOM when the timer finishes. Selenium can handle this using the above template:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec

driver = webdriver.Chrome()
driver.get('http://www.webcountdown.net/?c=3')  # Starts a 3 second timer.
if WebDriverWait(driver, 5).until(ec.presence_of_element_located((By.ID, 'popupiframe'))):
    print('Popup located!')

The above code should open the countdown page, run a three second countdown and print "Popup located!" after the three second countdown completes. This works because WebDriver is told to wait for up to five seconds for the popup to appear.

If, for example, this were modified with a two second timeout for the WebDriverWait class, Selenium would raise a selenium.common.exceptions.TimeoutException because the timer does not finish (and therefore does not create the element with ID "popupiframe") before the two seconds are up.

What is WebDriverWait good for? Briefly - single page apps (SPAs).

Testing may require traversing an app's navigational elements and if the page is not fully reloading, Selenium will need to use WebDriverWait to do things like wait for a new section or table of data to load after an AJAX-style API call.

Other expected conditions will follow pretty much the same syntax and mostly have (very) verbose names. Two of the others that I have found useful in practice are text_to_be_present_in_element() and element_to_be_clickable().

Time waits

Lastly, I have also used a workaround method to do simple, explicit time-based waits without any expected conditions. One area where this happened to come in handy for me is testing the result of a Javascript-based "stop watch" that updates in real time. As part of a test, I initiate the stop watch, wait for two seconds and then verify the displayed time to be correct. To achieve this, I created a method that essentially does an expected conditional wait that times out intentionally:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait

def wait(self, seconds):
    try:
        WebDriverWait(self.driver, seconds).until(lambda driver: 1 == 0)
    except TimeoutException:
        pass

This method can be used, for example to wait five seconds by calling wait(5). WebDriverWait will raise an exception after five seconds because the until() argument is a simple lambda that will always return False. By catching and passing on the exception, this method just waits for the specified number of seconds and nothing else. Handy!

Improving Consistency with SauceLabs

These basics are enough to get things going in Selenium, but over time as test complexities increase and multiple developer environments evolve, consistency will become a considerable pain. In our experience developing Timestrap, there were inconsistencies causing test failures based on development OS (Windows, OS X, Linux flavors, etc.), web drivers (Firefox, Chrome, gecko, etc.), and seemingly the phases of the moon.

After trying many different things to stabilize environments, we eventually found and started using SauceLabs. SauceLabs provides a number of services related to testing and a few free tiers for open source projects, including Cross Browser Testing. Using this service can help bring stability and consistency to Selenium tests regardless of the local development environment.

To get started, SauceLabs requires an existing, publicly accessible open source repository (e.g. on GitHub, GitLab, etc.). Use the OSS Sign Up page with the "Open Sauce" plan to get started. Once signed up and logged in, there are a couple of different ways to take advantage of SauceLabs testing:

Manual Tests

If you have an Internet accessible project available, Manual Tests can be used to poke around and get a feel for the various environments supported. This can serve as a wonderfully quick and easy way to do some prodding from a virtual browser in iOS, Android, OS X, Windows, Linux using various versions of Safari, Chrome, Firefox, Internet Explorer and Opera. Once a session is complete, the dashboard will have a log with screenshots and videos available to view or download.

Automated Tests

While manual testing is quick and convenient, automated testing is the important feature necessary to improve the consistency of Selenium tests in Python overall. Running Python's Selenium tests through SauceLabs requires three key things:

Username and Access Key
WebDriver.Remote
Sauce Connect Proxy Client

Username and Access Key

From a logged in SauceLabs account, the access key can be found on the User Settings page. This key and the associated username will need to be available in the local test environment in order to execute the Selenium-driven tests on SauceLabs.

I recommend getting used to using the environment variables SAUCE_USERNAME and SAUCE_ACCESS_KEY as these will be used by the Sauce Connect Proxy Client for local development testing.

On Linux this can be achieved with:

export SAUCE_USERNAME={sauce-username}
export SAUCE_ACCESS_KEY={sauce-access-key}

WebDriver.Remote

Selenium provides a WebDriver.Remote class for interacting with a command-based remote server running the WebDriver protocol. The class must be initialized with two arguments, command_executor, a URL pointing to the remote command point, and desired_capabilities, a dictionary of settings for the executor.

For SauceLabs, the command_executor should be set to http://SAUCE_USERNAME:SAUCE_ACCESS_KEY@ondemand.saucelabs.com/wd/hub where SAUCE_USERNAME and SAUCE_ACCESS_KEY represent the properties outlined in the previous section of this post.

The desired_capabilities dictionary is used to provide the environment settings to SauceLabs. SauceLabs has a wonderful Platform Configurator tool for easily selecting from the available options.

To use the example below, the local environment must provide two variables: SAUCE_USERNAME and SAUCE_ACCESS_KEY. With these variables set, the following code will create a remote WebDriver set up to access SauceLabs using Chrome 48 on a PC running Linux:

import os
from selenium import webdriver

# Get the user name and access key from the environment.
sauce_username = os.environ['SAUCE_USERNAME']
sauce_access_key = os.environ['SAUCE_ACCESS_KEY']

# Build the command executor URL.
url = 'http://{}:{}@ondemand.saucelabs.com/wd/hub'.format(
    sauce_username, sauce_access_key)

# Build the capabilities dictionary (from Platform Configurator).
caps = {'browserName': "chrome"}
caps['platform'] = "Linux"
caps['version'] = "48.0"

driver = webdriver.Remote(command_executor=url, desired_capabilities=caps)
driver.get('https://www.google.com')
driver.quit()

After executing the above sequence, the SauceLabs Dashboard should show a new job with video and screenshots of Chrome on Linux loading Google. Neat!

When using Chrome, a chromeOptions dictionary can also be provided in the desired_capabilities dictionary with some more specific settings. Within that dictionary, a prefs dictionary can also be used to set further preferences. For instance, if testing needs to be done on an app that requires login, it may be helpful to use this chromeOptions dictionary:

caps['chromeOptions'] = {
    'prefs': {
        'credentials_enable_service': False,
        'profile': {
            'password_manager_enabled': False
        }
    }
}

Very simply, this prevents the "Do you want to save your password?" sort of dialog box from appearing in all screenshots of a test session after login.

Sauce Connect Proxy Client

All of this works great just as described... if the app being tested happens to be available on the public Internet. If that is not the case (and it probably isn't), SauceLabs provides the Sauce Connect Proxy to connect to your local app.

On Linux, for example, the proxy client can be installed like so:

wget https://saucelabs.com/downloads/sc-4.4.9-linux.tar.gz
sudo mkdir -p /usr/local/bin/
tar xzf sc-4.4.9-linux.tar.gz
mv sc-4.4.9-linux/bin/sc /usr/local/bin/
sudo chmod +x /usr/local/bin/sc
sc --version
#Sauce Connect 4.4.9, build 3688 098cbcf -dirty

The sc command will make use of the SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables. When executed with no parameters, the proxy client will run through some initialization leading to the message, Sauce Connect is up, you may start your tests. From here the client will simply sit and listen for commands and the SauceLabs Tunnels page should show the client as active.

With all of this in place, tests against a local development server can now be proxied up to SauceLabs and run in a considerably more consistent environment!

This method significantly improved the test infrastructure for Timestrap and allowed us to refocus on development instead of testing.

Putting It All Together

We can bring all this together in one (admittedly somewhat complex) Python test file:

import os
import threading
import time
import unittest
from http.server import BaseHTTPRequestHandler, HTTPServer
from selenium import webdriver
from selenium.webdriver.common.by import By


class TestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()
        self.wfile.write(b'<html><head><title>Python Selenium!</title></head>')
        self.wfile.write(b'<body><div id="main">Hello!</div></body>')
        self.wfile.write(b'</body></html>')


class TestRequest(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        server = HTTPServer(('127.0.0.1', 8000), TestHandler)
        cls.server_thread = threading.Thread(target=server.serve_forever, daemon=True)
        cls.server_thread.start()
        time.sleep(1)

        sauce_username = os.environ['SAUCE_USERNAME']
        sauce_access_key = os.environ['SAUCE_ACCESS_KEY']
        url = 'http://{}:{}@ondemand.saucelabs.com/wd/hub'.format(
            sauce_username, sauce_access_key)
        caps = {'browserName': "chrome"}
        caps['platform'] = "Linux"
        caps['version'] = "48.0"
        cls.driver = webdriver.Remote(command_executor=url, desired_capabilities=caps)

    @classmethod
    def tearDownClass(cls):
        cls.driver.quit()

    def test_request(self):
        self.driver.get('http://127.0.0.1:8000')
        self.assertEqual('Hello!', self.driver.find_element(By.ID, 'main').text)


if __name__ == '__main__':
    unittest.main()

TestHandler.do_GET() is a very simple method for http.server that returns the following HTML:

<html>
<head>
    <title>Python Selenium!</title>
</head>
<body>
    <div id="main">Hello!</div>
</body>
</html>

TestRequest.setUpClass() does three import things before running the tests:

Establishes the HTTPServer instance.
Starts the HTTP server in a thread (to prevent blocking).
Establishes the WebDriver.Remote instance using SauceLab as the command executor.

TestRequest.tearDownClass() simply shuts down the web driver.

Lastly, TestRequest.test_request() is the single test in this "suite". It simply loads the test server index page and asserts that the text "Hello!" is present inside div#main (which it should be).

Let's give it a try! Remember to set the SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables, first:

export SAUCE_USERNAME={sauce-username}
export SAUCE_ACCESS_KEY={sauce-access-key}
python tests.py
#E
#======================================================================
#ERROR: test_request (__main__.TestRequest)
#----------------------------------------------------------------------
#Traceback (most recent call last):
#[...]
#selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"main"}
#  (Session info: chrome=48.0.2564.97)
#  (Driver info: chromedriver=2.21.371459 (36d3d07f660ff2bc1bf28a75d1cdabed0983e7c4),platform=Linux 3.13.0-83-generic x86)
#[...]

Oh no! What happened? The important bit in the traceback is this: Message: no such element: Unable to locate element: {"method":"id","selector":"main"}. For some reason, Selenium was not able to find the div#main element. Since this test ran in SauceLabs, the SauceLabs Dashboard has information and a replay of the test session which reveals... oh... SauceLabs was trying to access the local network (127.0.0.1) and we forgot to start the proxy client. Oops!

Let's try that one more time, this time starting up the Sauce Connect Proxy (sc) before running the tests...

export SAUCE_USERNAME={sauce-username}
export SAUCE_ACCESS_KEY={sauce-access-key}
sc &
#[...]
#Sauce Connect is up, you may start your tests.
python tests.py
#[...]
#.
#----------------------------------------------------------------------
#Ran 1 test in 6.026s
#
#OK

Note: Don't forget to kill the sc process with, for example, pkill -x sc.

Hooray! This time the test ran successfully because SauceLabs was able to use the proxy client to access the local test server.

This means that the local development environment can still be used for testing without having to deploy between tests.

There you have it. With a local test server up and running, getting consistent results from Selenium can be incredibly smooth and save many, many testing headaches as the code base and developer contributions expand (hopefully!).

Discussion at Hacker News

Buying Lotion on Amazon.com

2017-06-11T13:45:00-04:00

I was recently shopping online for a specific type of lotion (my skin seems to hate all other types). I am somewhat predisposed to avoid Amazon.com because its size and increasing dominance of online shopping concerns me as it moves closer and closer to "shopping" (in the sense that "searching" means Google to most people). However, my biggest reason for avoiding Amazon.com is simpler: it has become incredibly confusing to shop there. Searching for just about anything will yield thousands of results and it takes (me) a lot of effort to determine which one is appropriate. This can be a great thing in a lot of cases, but more often than not it feels more like navigating a minefield of deceitful listings than comparing competing products.

This lengthy post evaluates one example - Aveeno "Daily Moisturizing" lotion. As with most personal care products, when searching online I pull up a couple of websites to compare prices. In this case I pulled up Amazon.com last, searched for "Aveeno" and was impressed to see just what I was looking to compare (the 18 oz., non-SPF version) as the first result. I clicked through and, curious to figure out the price per ounce, looked to the "add to cart" area. What are are my options?

Sizes select list with 37 sizes.
Styles (??) select list with three styles.
"Subscribe & Save" (pre-selected) and "One-time purchase" radio buttons.
"Qty" select list.
"Delivery every" select list (2 months pre-selected).
"Subscribe now" button.
"Add to List" button.
Three "Add to Cart" buttons with different prices.

Well, this is going to be complicated...

Sizes

Let's get straight to it - there are 37 different "sizes" of this lotion, here is a look at all of them, sorted by price per ounce.

37 Flavors (of Lotion)
Size	Price	Units	Oz./unit	Total oz.	Price/oz.
2 Pack 20 ounces each	$17.68	2	20	40	$0.4420
18 Fluid Ounce	$8.99	1	18	18	$0.4994
12 Fluid Ounce (Pack of 6)	$52.53	6	12	72	$0.7296
12 Fluid Ounce	$9.32	1	12	12	$0.7767
8 Fluid Ounce (Pack of 3)	$23.05	3	8	24	$0.9604
1 Fluid Ounce (Pack of 36)	$34.92	36	1	36	$0.9700
12 Fluid Ounce (Pack of 2)	$27.56	2	12	24	$1.1483
2.5 Ounce (Pack of 2)	$5.92	2	2.5	5	$1.1840
18 Ounce (Pack of 3)	$99.99	3	18	54	$1.8517
4Pack (18 Ounce)	$140.16	4	18	72	$1.9467
2Pack (18 Ounce)	$70.11	2	18	36	$1.9475
5Pack (18 Ounce)	$176.25	5	18	90	$1.9583
3Pack (18 Ounce)	$108.19	3	18	54	$2.0035
5 Pack (6 Count 12 oz bottles)	$731.35	30	12	360	$2.0315
4 Pack (6 Count 12 oz bottles)	$585.13	24	12	288	$2.0317
2 Pack (6 Count 12 oz bottles)	$292.58	12	12	144	$2.0318
4Pack 20 ounces	$176.25	4	20	80	$2.2031
10Pack 20 ounces	$450.21	10	20	200	$2.2511
8Pack 20 ounces	$360.23	8	20	160	$2.2514
6Pack 20 ounces	$270.18	6	20	120	$2.2515
3Pack (3 Count 8 oz bottles)	$171.63	9	8	72	$2.3838
5Pack (3 Count 8 oz bottles)	$292.62	15	8	120	$2.4385
4Pack (3 Count 8 oz bottles)	$234.16	12	8	96	$2.4392
2Pack (3 Count 8 oz bottles)	$117.19	6	8	48	$2.4415
2Pack (12 Ounce)	$65.53	2	12	24	$2.7304
4Pack (12 Ounce)	$131.18	4	12	48	$2.7329
5Pack (12 Ounce)	$165.00	5	12	60	$2.7500
3Pack (12 Ounce)	$101.28	3	12	36	$2.8133
5Pack (36 Count 1 oz bottles)	$675.12	180	1	180	$3.7507
4Pack (36 Count 1 oz bottles)	$540.10	144	1	144	$3.7507
3Pack (36 Count 1 oz bottles)	$405.12	108	1	108	$3.7511
2Pack (36 Count 1 oz bottles)	$270.09	72	1	72	$3.7513
4Pack (2 Count 2.5 oz bottles)	$131.16	8	2.5	20	$6.5580
2Pack (2 Count 2.5 oz bottles)	$65.69	4	2.5	10	$6.5690
5Pack (2 Count 2.5 oz bottles)	$164.89	10	2.5	25	$6.5956
3Pack (2 Count 2.5 oz bottles)	$101.39	6	2.5	15	$6.7593
2.5 Ounce (Pack of 4)	$99.99	4	2.5	10	$9.9990

The cheapest option is not really surprising, a two pack of 20 oz. bottles comes out to $0.4420/oz. The default selection, one 18 oz. bottle, is only about five cents more at $0.4994/oz. The next entry jumps $0.23/oz. for, oddly, a six pack of 12 oz. bottles. Where is this cheapest item (the 18 oz. two pack) in the actual list? It's option #6 - the default is #30 (of course!), so it's only necessary to scan up the long list and (hopefully) click the right one. This option isn't sold from or shipped by Amazon.com, but is fulfilled by a third-party seller.

So, one can work all this out for each option - it took me about half an hour with Firefox's developer tools, a text editor with regex search/replace, and a spreadsheet - or take a wild stab and pay up to 2,160% extra (at $9.99/oz.) than the cheapest (non-subscription) option.

The variation and style of the "size" descriptions also is quite confusing in itself. Is a "2Pack" two bottles of lotion? Well, it depends. In one case it is indeed two, but in others it is four, six, or... 72! Having so many options is not necessarily a bad thing - perhaps someone is hosting a skin care event and wants to hand out mini bottles to hundreds of attendees - but the inconsistency in how each size is described is silly.

Note: in the time since I began typing this post (an hour or so ago), the number of size options increased to 38.

Styles

There are only three style options (phew!). They are:

Daily Moisturizing (the default)
Daily Moisturizing, SPF 15 (ok, that fits)
Lotion (what?)

Selecting Daily Moisturizing, SPF 15 redirects me to the 2Pack (12 Ounce) size for $65.53 ($2.7304/oz.). Is this the cheapest option for the style? Although it's not clear from the size names (surprise!), the list of options is in fact a mix of SPF and non-SPF. To Amazon.com's credit, the list does sort of indicate this with an "Available in (style)" note instead of price for relevant entries. Anyway, it appears that the 12 Fluid Ounce option is the cheapest at $9.32 ($0.7767/oz.) - 71% cheaper.

The Lotion style really simplifies things (in a way) - the default size is 2.5 Ounce (Pack of 4) for $99.99 ($9.99/oz. - aka the most expensive option in the list). If that seems a bit pricey, just pay the same amount for the only other size - 18 Ounce (Pack of 3) ($1.8517/oz.) - 440% cheaper, woo! Both of these "sizes" come, unsurprisingly, from the same third-party seller and begs the question - is this intentional deceit or a legitimate accident? Well, the seller has 99% positive feedback in the last 12 months on 5,938 ratings so... is that good? It sounds good!

Everything Else

Assuming no changes are made to the Size or Style options, that leaves nine other fields to consider to get this item in a cart. There are two primary options - "Subscribe & Save" or "One-time Purchase".

One-time Purchase

Selecting "One-time Purchase" simplifies things a bit, the "Subscribe now" button becomes an "Add to Cart" button and the default option is still the third best deal of all combinations.

Other Sellers on Amazon.com

It is worth noting that, if one does not select the "One-time Purchase" option, there are three "Add to Cart" buttons on the screen. For a frequent user of Amazon.com, it is probably clear enough what these options are (the same item from different third-party sellers). However, it doesn't seem much of a stretch to imagine a first-time shopper seeing and clicking the first available "Add to Cart" button. In this case, the first (and cheapest) option in that list is $13.09 ($0.7272/oz.). Hilariously, this is actually the third best price available (fourth if the "Subscribe & Save" option is considered). Not a bad deal for the confused shopper!

Conclusion

I do not believe that Amazon.com is trying to be deceitful with all this cruft. My best guess is that some third-party sellers try their best to game the system and others just have to try to keep up.

Amazon.com has the burden of policing this sort of activity and this is just one example of a very deep and serious problem - a massive amount of confusing options will drive people away.

Half way through typing this up, I realized why I took such a great interest in figuring out what the hell is going on with all the options. In a few months my wife will give birth to our first child and we will likely be going out of our minds (we already are a little bit!) trying to understand what we need to purchase and how to gauge quality and price. Amazon.com makes researching products very difficult and I will try my hardest to avoid using it when possible.

Bonus Conclusion!

Anyone needing to buy Aveeno lotion without SPF now has all the details! Although, given the addition of a 38th size option in the past few hours, I imagine the value of this information won't last much more than a week or month at best. Ultimately, I purchased elsewhere at a price of $0.3718/oz., and I didn't have to subscribe to scheduled future purchases.

Discussion at Hacker News

Yes This Is A Really Long Request URL

2017-04-20T10:00:00-04:00

Yesterday, while reviewing some logs I came across a curious entry in an Apache error log:

[Wed Apr 19 08:51:48.119666 2017] [core:error] [pid 29210] (36)File name
too long: [client 137.226.113.7:40907] AH00036: access to
/YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForR
esearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongReques
tURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALo
okAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurpos
eWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisI
sAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPu
rposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWe
AreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUs
erAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreSca
nningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyL
ongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePlea
seHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingI
tOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTH
XYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann failed
(filesystem path '[...]')

Formatted to plain English: Yes, this is a really long request URL but we are doing it on purpose. We are scanning for research purpose. Please have a look at the user agent. Thanks!

What does the user agent for this request have to say?

Here is the access log entry:

137.226.113.7 - - [19/Apr/2017:08:51:48 -0400] "GET [...] HTTP/1.1" 403
1471 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
Scanning for research (researchscan.comsys.rwth-aachen.de)"

The website referenced in the user agent, researchscan.comsys.rwth-aachen.de, explains that this request is part of a research project at RWTH Aachen University in Germany and 137.226.113.7 is indeed a part of the university's network.

Interestingly, this is the first time I have seen such a request in any web server log (I have had occasion to look through more than a few). The Wayback Machine has an archive of the user agent page from 6 Dec 2015, so it seems that the research has been going on for at least a year. I checked for the YesThisIsAReallyLongRequestURL string in (a lot of) logs for about 10 different websites of varrying size and did not find any other instances. I wonder how they determine which sites to scan...

Responding to Identity Theft

2017-03-23T22:00:00-04:00

On Sunday, 19 March around 10AM I received an email with the subject line IMPORTANT: ProtectMyID Surveillance Alert. I was busy at the time so I flagged the message and moved on with my day. I had received these alerts a few times before when applying for credit cards and getting a home loan. While I hadn't done either of those things lately, I still was not particularly alarmed by the subject alone and had many other things on my mind. Going about the busy day, I promptly forgot about the email.

Before starting the work day the next morning, I noticed and remembered the email in my inbox. Why do I even have ProtectMyId? Because my personal data was part of a breach at some point. Which breach? I don't even recall - there have been so many. I opened the email and logged in to find that a new credit account with a provider I had never used had been opened in my name. Surprise! I am a victim of identity theft.

What do I do now?

Credit Reporting Agencies

First, a bit of background - ProtectMyID is a service available from Experian, one of the three Credit Reporting Agencies (CRAs) responsible for monitoring and "scoring" credit for individuals (whether they know it or not). The USA.gov website defines a CRA like so:

A credit reporting agency (CRA) is a company that collects information about where you live and work, how you pay your bills, whether or not you have been sued, arrested, or filed for bankruptcy. All of this information is combined together in a credit report. A CRA will then sell your credit report to creditors, employers, insurers, and others. These companies will use these reports to make decisions about extending credit, jobs, and insurance policies to you.

Basically, these agencies collect, "score" and sell your personal information to companies and sell your own information back to you in the form of credit reports and identity "protection" services. Not surprisingly, the other two agencies (Equifax and TransUnion) have similar services.

Want to monitor your information at all three agencies? That'll be about $50 per month for basic service. In fact, upon logging in to check this new alert, the ProtectMyID website displayed a big warning box telling me that my service is about to expire and I will need to pay if I want to continue using it. Good timing!

Oh, a discounted rate!

An Attempt to Dispute

Anyway, on to (what I guessed to be) step one: dispute this new account on my credit report. Included with each alert on the ProtectMyID website is a big red Dispute Item button - great, this should be simple! I clicked the button for the fraudulent account and was redirected to a page where, for some reason, I had to click another button indicating that I wanted to dispute an item on my Experian report.

Next, I was asked for a "report number". Taking a look at the help on the page, I realized I did not have one because I didn't have an actual credit report, just the ProtectMyID alert. I navigated my way back to the alert page and found a "Credit Report" link. I clicked the link and was greeted with a blank page - no error, no content, just blank. Upon doing this I recalled the exact same thing happening multiple times in the past. At least once, I received an "error" message explaining that this was not part of the free service.

In fact, the actual credit report is not even part of the paid service, that costs extra!

Lacking an actual credit report, Experian provides another path - provide all my personal information and answer a series of questions about the contents of my report in order to access it and dispute the item. My options in this round, each with a multiple choice list of answers including a NONE OF THE ABOVE, included:

A current or previous phone number that is associated with you.
The last four digits of your primary checking account number.
A person you have lived with in the last 10 years.
A city that you have previously resided in.

The result?

Nope.

This has happened every single time I have attempted to verify myself to Experian in particular (Equifax and TransUnion have worked fine). I have not always been entirely sure of the answers, but in this case (for this blog post) I am actually 100% sure I answered them all correctly.

So this leaves me with two options: print and mail a letter to Experian and wait for the response or try something else. Let's try something else...

Calling Around

Included with the alert on ProtectMyID is a phone number for the reporting company (bank, creditor, etc.) so I decided to check that for the fraudulent record and reach out directly.

The "phone" entry for the new account alert: "BYMAILONLY". Helpful.

I scrolled through the rest of my alerts and found the same bank again, along with two other unknown entries! All three of these entries were for "hard" inquiries - meaning a company received my credit report at "my" request (see also: Hard vs. Soft Inquiries), only I didn't make any of the requests and all three occurred on February 20, nearly a month prior to my alert!

Why didn't I receive alerts about these hard inquiries from ProtectMyID? How has this terrible infrastructure failed this time? It didn't, actually. The emails went to an address I had recently migrated away from and I forgot to update my address with ProtectMyID. I checked the old email and, sure enough, all three alerts were there. Oops.

Bank #1

The hard inquiry alert for Bank #1, unlike the open account alert, does contain a phone number. I gave that number a call and quickly got connected to an agent who was very helpful - this was a nice relief when other avenues were failing me. The first thing the rep asked for was my full name, then my address, then my social security number... this conversation gave me a bit of pause, but ultimately it's understandable that all this information is needed in order to verify my identity. Nonetheless, it left a bad taste in my mouth to be providing this sort of information over the phone in response to its being compromised and used fraudulently.

It turned out that the account was opened, but immediately flagged for suspicious activity and, luckily, no charges were made with the account. The rep closed the account, initiated an internal investigation and provided me with some ideas for next steps. Overall very positive.

Bank #2

Next, I called the number for Bank #2 associated with its hard inquiry. Again, I was able to get to a rep pretty quickly. Again, I was asked to provide my name and address. However, in this case I was only asked for the last four digits of my social security number. This was at least a bit more comforting. The rep found the fraudulent application, denied it, started an internal investigation and provided me with a reference number. Quick and easy.

Bank #3

Here again no phone number is provided with the alert, so I went searching for a number on the bank's website. Interestingly, they have a page dedicated to identity theft which basically says "Don't call us, call the FTC". Undeterred, I dug up another number, gave them a call and quickly discovered that they have no interest in to talking to anyone other than account holders...

After about five minutes of responding to the same prompts with the same answers and pounding keys, I managed to get an actual person on the phone. Here again I had to provide all the same identifying information to get the issue addressed. The rep found and closed the application, but did not say anything about investigating or provide any reference number.

Getting My Credit Reports

Everyone is entitled to a free credit report once per year (how lucky!). I was not at all surprised to find that searching for "free credit report" online turns up more than a few phonies. The actual website associated with the law is AnnualCreditReport.com.

I walked through the process and had relatively little trouble. Equifax and TransUnion both provided sane verification questions and allowed me to download the report immediately. Experian, on the other hand, once again provided some odd questions and was not able to verify me. Apparently, I will receive a letter and code (to access my report) via the mail within three weeks. Three weeks.

I reviewed the Equifax and TransUnion reports closely and, despite finding a number of things that surprised me, did not find any additional fraudulent entries.

While I am relieved that I was able to retrieve two of the three reports and that nothing additional was found, I find it rather perplexing that this information is kept so secret from the people that it "rates". The fact that these reports can only be retrieved once a year and otherwise must be paid for is terrible. If I had not happened to have a service provided because of a breach at just the right time, it may have taken me years (and potentially a denied credit application or worse) to find this or more serious damage.

While these Credit Reporting Agencies maintain and "score" such deeply personal and potential harmful information, protective services should be provided to individuals free of charge.

Credit Report Protections

With all this information in hand, I began thinking about the future and wondering how on Earth I can prevent having to go through this headache again. Eventually, I found that there are a couple of options to protect a credit report in various ways. While not exactly what I was thinking, it seemed like a good enough start. There are generally three options:

1. Fraud Alerts

This is a 90-day alert that adds a requirement of placing a phone call with a number associated with the alert to verify any new inquiries or accounts. It can fall off or be renewed after 90 days. This also, apparently, entitles you to a free credit report, but I did not test this. (Fraud Alerts)

2. Extended Fraud Alerts

This alert goes quite a bit further. It lasts seven years, entitles you to two free reports from each agency and takes your name off pre-screened offer marketing lists for five years. Each agency must be contacted separately to enable this alert and it may require additional paperwork and proof of actual identity theft. (Extended Fraud Alerts)

3. Credit Freezes

This most serious of the three options, a credit freeze prevents almost all access to a credit report. The durations and rules for credit freezes vary from state to state and, unlike fraud alerts, most credit freezes require paying a fee (unless you have a police report, in some cases). Generally, except for state-imposed limits, this type of protection lasts until you have it lifted. (Credit Freezes)

Placing a Fraud Alert

I chose to place an initial 90-day fraud alert because the credit freeze option seemed a bit extreme and the requirements of the extended alert, as indicated by Experian, seemed daunting. I used Experian's Fraud Alert Center to walk through instructions that eventually gave me a phone number to call. The number was a fairly straight forward automated system that took a few pieces of information, supplied me with a confirmation code and set the alert.

How does this appear on a credit report? On my TransUnion report, in the form of this simple note:

SECURITY ALERT: Initial Fraud Alert: Action may be required under FCRA before opening or modifying an account. Contact consumer at (xxx) xxx-xxxx. (Note: This alert is set to expire in 07/2017.)

Why do I know this? Despite setting the fraud alert on 20 March, I received a new ProtectMyID alert email noting the addition of the same fraudulent account to my TransUnion report. While I can accept some amount of lag between systems, it greatly concerns me that this addition was not prevented three days after I placed fraud alert.

The lack of urgency, shoddy technical systems and poor usability of these CRA tools make the industry seem entirely out of touch with the feeling of urgency I suspect most people have when something like this happens.

Alerting the FTC

As my final step (for now), I created an account and filed a report at IdentityTheft.gov. The guided questions leading to the sign up form were actually quite useful in helping me remember which breaches my data has been caught up in.

Side note - I attempted to verify one breach that I wasn't sure about and was rather disappointed when, after submitting all my info (yet again), I landed on a simple page explaining that I would receive a response by mail within some weeks.

Once signed up and logged in, the site provides a detailed checklist-style tool for walking through all the basic response steps. Many of these things I had done already, but I was happy for some extras such as links relating to protection services offered by specific breached companies and ready-to-send form letters for banks and other institutions.

Ultimately, this was a vital tool and probably would have been a better starting point for the entire ordeal. One particularly interesting fact I picked up from reviewing things with this tool is that the identity theft "report" provided should be all I need to add an extended alert to my credit report. The FTC puts it rather simply:

If you’ve created an Identity Theft Report, you can get an extended fraud alert on your credit file.

Compare that statement to this one from Experian's Fraud Alert Center:

If you are a victim of identity theft and submit a copy of a valid identity theft report that you have filed with a Federal, State or local law enforcement agency, then you may request an Extended Fraud Victim Alert, which lasts for 7 years.

As it turns out, the FTC is a federal law enforcement agency, so Experian's statement is essentially true but perhaps misleading. If the IdentityTheft.org report (which is basically a letter of attestation) is truly all that is needed - why not simply link to the site? The statement as it stands would likely deter anyone who hasn't suffered significant harm and already filed a local police report.

Next Steps

I am currently waiting for two items in the mail:

My Experian credit report
A confirmation letter from a large breach

Once those items arrive, I will review and take any necessary additional steps. I will also probably make use of the extended fraud alert. Some information in my Experian report is necessary for that so I have some time to think it over.

If I were doing this all over again (and hey, maybe I will!), I would follow these steps in this order:

File a report at IdentityTheft.gov.
Call the involved banks (or other institutions).
Get a credit report from each CRA (either from AnnualCreditReport.com or by adding initial fraud alerts).
Add an extended fraud alert to each CRA's report.

Despite the fact that these agencies talk amongst each other and that the banks claimed they would repeal the fraudulent information, I am compelled to use every possible avenue to report and correct things. The lack of urgency displayed by the agencies in particular does not instill any confidence.

In the end, I am still left feeling quite vulnerable. My information is definitely out there - I already knew that and wasn't happy about it. But now it has been used, and thus confirmed, in the wild. This is information that does not expire and is not easily changed, and I'm supposed to be content with one year of free "protection" (which is just monitoring)?

All of the protective services, credit reports and scores offered by these agencies should be provided to the individuals they affect at no charge. If Experian, Equifax and TransUnion are to profit from such sensitive, private information by selling it to third parties, they must also bear the burden of monitoring and protection.

RDAP Explorer

2017-02-06T21:30:00-04:00

Having fallen behind a bit on Takeout Inspector, the 12 Years of Gmail series and some other projects, I decided to try to put something very simple together from beginning to end and actually launch it. One of my previous posts, Examining the Remnants of a Small DDoS Attack introduced me to the Python package ipwhois and the alternative WHOIS system RDAP. This eventually led me to a quick and simple project called RDAP Explorer...

What is RDAP?

According to APNIC

The Registration Data Access Protocol (RDAP) is an alternative to WHOIS for accessing Internet resource registration data. RDAP is designed to address a number of shortcomings in the existing Whois service. The most important changes are:

Standardization of queries and responses

Internationalization considerations to cater for languages other than English in data objects

Redirection capabilities to allow seamless referrals to other registries

The most important advantage of RDAP over WHOIS is the Standardization of queries and responses. While reviewing a large set of IP addresses, I found it rather difficult to deal with non-standard (and sometimes nonsensical) output of WHOIS queries. Mostly they were easy enough to parse, but the odd balls made the process annoying and time consuming.

Enter ipwhois

Eventually I found my way to the wonderfully useful ipwhois package and this made dealing with a large set of IP addresses considerably easier. I dug in to the code a little bit to see just how this was being done and eventually learned about the aforementioned virtues of RDAP. Here is an example of the type of output ipwhois produces, for example, with one of Google's ever-popular public DNS servers -

import ipwhois
IP = ipwhois.IPWhois('8.8.8.8')
result = IP.lookup_rdap()
print result
{'raw': None, 'entities': [u'GOGL'], 'asn_registry': 'arin', 'network': {'status': None, 'handle': u'NET-8-8-8-0-1', 'name': u'LVLT-GOGL-8-8-8', 'links': [u'https://rdap.arin.net/registry/ip/008.008.008.000', u'https://whois.arin.net/rest/net/NET-8-8-8-0-1', u'https://rdap.arin.net/registry/ip/008.000.000.000/8'], 'raw': None, 'country': None, 'ip_version': u'v4', 'start_address': '8.8.8.0', 'notices': [{'description': u'By using the ARIN RDAP/Whois service, you are agreeing to the RDAP/Whois Terms of Use', 'links': [u'https://www.arin.net/whois_tou.html'], 'title': u'Terms of Service'}], 'end_address': '8.8.8.255', 'remarks': None, 'parent_handle': u'NET-8-0-0-0-1', 'cidr': '8.8.8.0/24', 'type': None, 'events': [{'action': u'last changed', 'timestamp': u'2014-03-14T16:52:05-04:00', 'actor': None}, {'action': u'registration', 'timestamp': u'2014-03-14T16:52:05-04:00', 'actor': None}]}, 'objects': {u'GOGL': {'status': None, 'roles': [u'registrant'], 'handle': u'GOGL', 'entities': [u'ABUSE5250-ARIN', u'ZG39-ARIN'], 'links': [u'https://rdap.arin.net/registry/entity/GOGL', u'https://whois.arin.net/rest/org/GOGL'], 'raw': None, 'notices': None, 'contact': {'kind': u'org', 'name': u'Google Inc.', 'title': None, 'phone': None, 'role': None, 'address': [{'type': None, 'value': u'1600 Amphitheatre Parkway\nMountain View\nCA\n94043\nUNITED STATES'}], 'email': None}, 'events_actor': None, 'remarks': None, 'events': [{'action': u'last changed', 'timestamp': u'2017-01-28T08:32:29-05:00', 'actor': None}, {'action': u'registration', 'timestamp': u'2000-03-30T00:00:00-05:00', 'actor': None}]}}, 'asn_country_code': 'US', 'asn_date': '', 'asn_cidr': '8.8.8.0/24', 'nir': None, 'query': '8.8.8.8', 'asn': '15169'}

With a bit of formatting, the end result looks quite nice and is easy to work with (in code):

{
   'raw':None,
   'entities':[
      u'GOGL'
   ],
   'asn_registry':'arin',
   'network':{ ... },
   'objects':{ ... },
   'asn_country_code':'US',
   'asn_date':'',
   'asn_cidr':'8.8.8.0/24',
   'nir':None,
   'query':'8.8.8.8',
   'asn':'15169'
}

However, outside of code this is not particularly easy to parse (well, still better than WHOIS results), particularly when digging deeper in to the data with some queries producing hundreds of lines of information. There are many, many whois services available online, but I could not find a comparable service for RDAP. So I decided to try my hand at making (and, more importantly, launching) one...

RDAP Explorer

The fruit of my (light) labor is RDAP Explorer, a web interface for running RDAP queries using the ipwhois package. Currently, the output is not much more than a nicely formatted expandable tree of information. This works well for containing the vast amounts of data that RDAP queries can produce.

The website is my first project built on Django, which was pretty painless to get started with. This also enabled me to continue learning about nginx and introduced me to uwsgi. This combination proved a bit tougher to get a handle on given my greater experience with Apache and PHP, but ultimately getting everything running in "production" was not particularly difficult or time consuming.

For future development I intend to add clearer summary information for the results and searching of the full result tree. For now I will just enjoy my first launch of a Python project.

12 Years of Gmail, Part 5: Mail

2016-12-05T12:08:00-05:00

This post is part of my series, 12 Years of Gmail, taking a look at the data Google has accumulated on me over the past 12 years of using various Google services and documenting the learning experience developing an open source Python project (Takeout Inspector) to analyze that data.

After taking a look at the chat data in my export, I am finally ready to move on to some of the actual mail! Much of what I will look at here is pretty similar to what I was able to turn up with chat data. I tried to branch out a bit, bringing in a new package to create word clouds, and also refactored some of the Takeout Inspector code to form the beginning of a more "formal" report generating process (instead of just spitting out a single HTML file with only a certain subset of the data). Hopefully I can continue to improve this to a point allowing for easier report generation for any user. Anyway, on to the mail data!

Top 10 Recipients

This is one of the first graphs I produced while prototyping this idea. The top recipient (person I have sent email to) by far, Crystal Meyer, is my wife and the remaining folks are a mix of friends and family so there is nothing much to dig in to for this graph.

Top 10 Senders

This is at least slightly more interesting - while Crystal Meyer is the top person I send email to, she is not the top person I receive email from. So who is this mystery Lindsey Campbell? Not a person at all! This happens to be a "noreply" style address used for automated emails from a website forum dedicated to a Left 4 Dead game server that I operated for a couple of years.

Number two on this list, Craig Wilber is also not a real person. This heavy traffic is from an automated email service from DNSstuff that I have used in the past to alert me of DNS issues for a website. For exactly one year from 2006 to 2007 I received a couple of emails from this service every day. Looking at the content of some of these emails, I am not quite sure why I received so many for such a long time. They all report odd results about changing DNS records every couple of hours. Curious...

After those two and my wife, the rest of the list rounds out with a mix of more automated email services and other friends and family members as above.

Thread Durations

This graphs uses Gmail's thread IDs to determine total duration for each mail thread (based on the Date header of the first and last email). The following query retrieves the basis for this information:

SELECT strftime('%s', MAX(`date`)) - strftime('%s', MIN(`date`)) AS duration,
COUNT(message_key) AS message_count
FROM messages
WHERE gmail_labels NOT LIKE '%Chat%'
GROUP BY gmail_thread_id
HAVING message_count > 1;

In this case I chose to only include threads of more than one email (Line 5) to keep out things like newsletters and other automated email services. So this should reflect mostly real conversations with actual people.

No duration really dominates here, but if this were re-categorized as "one week or less" and "more than one week", the latter would only account for a little more than 10% of communications.

The longest thread lasted more than seven years (221,932,765 seconds) and only had four emails. It was an alert service for Bug 285774, an issue with the Camino web browser that I was apparently interested in back in 2006. In fact, the top five longest threads are all Mozilla bug reports relating to Camino (aka not real conversations).

The first and longest real thread is only three emails wherein I send my résumé to someone and receive a response nine months later. I guess he wasn't impressed...

Thread Sizes

Again using Gmail's thread ID email header, this graph compares thread sizes by the total number of emails in the thread (at least two emails). The curve ends up being almost exactly inverse except for small deviations and a couple of anomalies leading up to one big 98-email thread.

The first small anomaly is nine 23-email threads (zoom in to see it). Not surprisingly, these threads have nothing to do with one another. Some are between me and the developer of Quinn back in the early 2000's, another is a vacation planning thread with some friends, one relates to my wife and I closing on our first home, and the remainder are random conversations with friends.

The more noticeable anomaly is 19 61-email threads and it turns out these threads are in fact related somehow. As I mentioned previously in the Top 10 Senders section, I used to run a forum for a Left 4 Dead game server. This forum was not hugely popular, but it was eventually targeted by automated spam service that would attempt to create large swaths of accounts to spam the forum. In order to combat this, I activated a function requiring manual review of all newly created accounts. This is what led the noreply address from the forum to be far and away the top sender to my account.

Why 61 emails per thread? I'm not sure. My best guess was that Gmail grouped large threads after a set amount of time, but these threads range from about two to seven days with no one thread having the same duration. I continued digging around looking for similarities and, finding none, turned to the Internet. While I could not find any definitive source or reason for this grouping, at least a few other folks have stumbled on the oddity [1, 2]

There are only two more numbers of note after 61:

Two 66-email threads which are unrelated but both about planning and launching a website.
One 98-email thread, once again about the development process for the Quinn website.

Activity by Day of the Week

The distribution here for the days is not terribly surprising as I have never done much emailing on the weekends. But I am surprised to see the sent and received number almost 50/50 for every day of the week. I suspect this may be skewed a little bit because spam is cleared out automatically every 30 days and because I (regrettably) "cleaned" out a lot of old email before doing my Takeout export. Had I not cleaned things up, I suspect the sent/received distribution would be closer to 40/60 than 50/50.

Activity by Hour of the Day

This activity falls pretty squarely in line with my chat time activity. Apparently the best time to catch me for a response of any kind is on a weekday at/around EST lunch time (11AM - 1PM EST).

Interestingly, I think both the day of week and time of day activity were a habit of Gmail in particular. Since I moved away from it, I find that I am more prone to check and respond to emails in the evenings and on weekends. Perhaps this is simply because of the lack of chat functionality with my new email provider. I wish that I had had a longer history of Gmail before chat was introduced, as I wonder if that addition made some larger effect on my overall communication habits.

Label Usage

I was never a particularly heavy user of labels so this graph is mostly dominated by the default labels that Gmail applies automatically. Removing those from the graph (click the labels in the legend) reveals the labels I did actually use:

The most used label, FBF refers to Friends of Burkina Faso, an organization I provide some volunteer work to on occasion.
PC labeled emails relate to my time in Peace Corps, most of these emails are part of the long application process.
House emails relate to the many, many back-and-forth emails and documents involved in purchasing a home.
RAID refers to the Rogue AI Dungeon project I worked on with friends.
CHLHS is the Chesapeake Chapter of the U.S. Lighthouse Society, another organization I have volunteered with in the past.
Last but not least, haxors.com is a domain that I have owned for many years and put a lot of random projects on. By the time I started using Gmail, I used this domain less and less so there are not very many emails with this label (compared to the others).

I never found the labels concept particularly useful - my emails tend to have one primary subject and mixing labels would be a very rare occurrence. Prior to using Gmail and labels, I did make heavy use of folders with my own email services and to this day I still do with Exchange-based services. The Gmail experience did a very effective job of pushing users to the "archive it all and search as needed" philosophy thanks to Google's powerful search algorithms.

Subject Word Cloud

Most of the words that pop out here are not at all surprising given the topics covered in this post. The subject line of that automated forum email was always the same, "Activate user account", so it dominates the graphic. The rest tend to be either common English parts of speech and things related to Peace Corps or side projects I have worked on.

This graphic is generated using the wordcloud package, which has a very simple API for Python and also includes an advanced feature for building the word clouds using masks.

As this graphic is only based on email Subject headers, getting the data from Takeout Inspector's sqlite table is very simple:

SELECT subject FROM messages
WHERE gmail_labels NOT LIKE '%Chat%' AND subject != '';

From here Python is used to separate and clean the words and calculate frequency:

words = {}
for row in c.fetchall():
    subject = row[0]
    for prefix in ['Re:', 'Fwd:']:
        subject = subject.replace(prefix, '')

    subject = re.sub('[^a-zA-Z. ]', '', subject).strip().lower()

    for word in subject.split(' '):
        if word:
            word = word.rstrip('.')
            if word not in words:
                words[word] = 0
            words[word] += 1

Lines 3 & 4 remove common email subject prefixes.
Line 6 filters out non-alpha characters, leaving dots (for web addresses) and space characters, and converts all letters to lower case.
Lines 8 - 13 break each subject in to a list of words, removes periods from the end of words with rstrip (this assumes sentence structure) and finally ticks the frequency count for each word.

If required, the sorted() function is great for creating sorted lists from a dictionary like words above. However this word cloud simply filters the list for all words with more 100 occurrences before passing the data on to wordcloud's generate_from_frequencies() method. Wordcloud can also handle the frequency calculate itself with the regular generate() method

More to Come

Much of the graphs generated for email information are very similar to what was done for chat information. I will use a future post in the series to dig deeper in to some of the more technical email headers, hopefully take a look at body word usage/statistics and come up with some other ways to look at and compare more aggregate message data.

Examining the Remnants of a Small DDoS Attack

2016-12-03T12:00:00-05:00

On Sunday (27 November 2016) a small website that I advise on was the victim of a DDoS attack that managed to knock the site offline. I received notice on Monday that the website was not working. I was able to ssh to the web server and quickly found that the database service was stopped. After a brief examination of the database logs (nothing too out of the ordinary), I started the service back up and sure enough the website came back online. As the website runs on Drupal, I logged in to take a peak at the Recent log messages and found hundreds of records of log in attempts from a lot of different IP addresses. User accounts on the website are only used by administrators to update content, so it was clear that the site was hit by a DDoS attack!

After getting things back online, I poked around the various log files to try to get an idea of what happened. The Drupal watchdog logs seemed to indicate that the attack started around 15:22 EST and overloaded the server's memory around 15:42 EST. The Apache server's access logs, however, revealed that the attack started closer to 14:54 EST and lasted until 20:57 EST (about six hours). The Drupal watchdog service relies on a database connection, so because that was knocked offline around 15:42 EST, it was, of course, unable to continue recording the events.

Setup

I wanted to evaluate Apache's access logs a little closer, so I imported the logs in to an sqlite table. The access logs are in the Combined Log format: %h %l %u %t "%r" %>s %b "%{Referer}" "%{User-agent}" (host ident user time "request" response-code response-size "referer" "user-agent"). In order to turn this in to a tab-delimited file for import in to sqlite, I used the following regular expression:

(\d+\.\d+\.\d+\.\d+) (.+) (.+) \[(.+)\] "(.+)" (\d{3}) (\d+) "(.+)" "(.+)"

This regular expression matches each of the relevant pieces of information in to groups that can be used in a replacement expression using your favorite regular expression parser. In my case, I discarded the ident and user fields as they were always empty, further separated the request fields in to three parts (method, request and protocol) and imported all this data in to a table with the following structure:

CREATE TABLE "access"
(
    host TEXT NOT NULL,
    time TEXT NOT NULL,
    request_method TEXT NOT NULL,
    request TEXT NOT NULL,
    request_protocol TEXT NOT NULL,
    status_code INTEGER NOT NULL,
    response_size INTEGER NOT NULL,
    referrer TEXT NOT NULL,
    user_agent TEXT NOT NULL
)

Requests

First, taking a look at the total effort:

SELECT
    request_method AS method,
    COUNT() AS requests
FROM access
WHERE time >= '2016-11-27 19:54:55'
    AND time <= '2016-11-28 02:56:14'
GROUP BY request_method

method  requests
----------------
GET     55969
POST    9660

The time constraints here represent the first and last requests that were clearly a part of the attack. There was some legitimate traffic in this period but it is far outweighed by the attack traffic. This request reveals that there were 55,969 GET requests and 9,660 POST requests during the 6 hours of the attack. This averages out to about 180 requests per minute, or not very much at all.

But a more fine grained query reveals the timing a little better:

SELECT
    strftime('%d', time) AS day,
    strftime('%H', time) AS hour,
    strftime('%M', time) AS `min`,
    COUNT() AS requests
FROM access
WHERE time >= '2016-11-27 19:54:55'
    AND time <= '2016-11-28 02:56:14'
GROUP BY day, hour, `min`
ORDER BY requests DESC

day hour    min requests
------------------------
27  20      41      4230
27  20      39      2550
27  20      35      2448
27  20      38      2418
27  20      40      2346
27  20      37      2232
27  20      34      1980
27  20      44      1872
27  20      45      1872
27  20      36      1836
[...]

Between 20:41 and 20:42 (UTC) the server was actually hit with 4,230 requests (about 70 requests per second). This burst is ultimately what knocked the database server offline by 20:42 (the 15:42 EST noted above).

How much bandwidth did all of these requests chew up?

SELECT SUM(response_size) as bandwidth
FROM access
WHERE time >= '2016-11-27 19:54:55'
      AND time <= '2016-11-28 02:56:14'

bandwidth
---------
311446609

That's only about 300MB. Consistent with the size of the attack, not very much. It is definitely a lot more than the website in question normally does, but not enough to raise any eyebrows.

Devices

How many devices were involved in this attack?

SELECT COUNT(DISTINCT host) AS hosts
FROM access
WHERE time >= '2016-11-27 19:54:55'
      AND time <= '2016-11-28 02:56:14'

hosts
-----
311

Apparently only about 300. This is a far cry from the tens or hundreds of thousands of devices believed to be controlled by the news-making Mirai botnet, but apparently more than enough to defeat a $5/mo VPS instance with a non-optimized Drupal 8 installation.

Next up, which hosts did the most leg work?

SELECT host, COUNT() as requests
FROM access
WHERE time >= '2016-11-27 19:54:55'
      AND time <= '2016-11-28 02:56:14'
GROUP BY host
ORDER BY requests DESC

host            requests
------------------------
213.251.182.115     3738
212.114.109.218     2784
138.201.151.94      2448
213.251.182.111     2352
213.251.182.105     1992
213.251.182.106     1734
202.67.9.42         1554
184.106.10.128      1368
213.251.182.114     1338
103.45.230.202      1332
[...]

After the top ten, things tend to wind down rather slowly to round out the 311 hosts involved. So what can be found out about these hosts? Not a ton, really, other than who each IP belongs to. There are lots of bulk IP whois services available online, I choose a random one and got the following information:

Domain	IP	ISP	Organization	Country
gw-cluster015.ovh.net.	213.251.182.115	OVH SAS	OVH SAS	FR
server2.duurzaammedia.nl.	212.114.109.218	Whitelabel Hosting Solutions	Whitelabel Hosting Solutions	NL
static.94.151.201.138.clients.your-server.de.	138.201.151.94	HOS-181062	HOS-181062	DE
gw-cluster011.ovh.net.	213.251.182.111	OVH SAS	OVH SAS	FR
gw-cluster005.ovh.net.	213.251.182.105	OVH SAS	OVH SAS	FR
gw-cluster006.ovh.net.	213.251.182.106	OVH SAS	OVH SAS	FR
202.67.9.42	202.67.9.42	Unknown	Mollindo Company Jakarta	ID
fw-snet-n01.wc1.ord1.stabletransit.com.	184.106.10.128	Rackspace Hosting	Cloud Sites wc1.ord1	US
gw-cluster014.ovh.net.	213.251.182.114	OVH SAS	OVH SAS	FR
sd108202.server.idn.vn.	103.45.230.202	Online Solution Company Limited	Online Solution Company Limited	VN

Most of these devices appear to be hosted on OVH, and Rackspace is another big player represented here. This likely indicates that the devices are either compromised websites or rented machines set up specifically for DDoS'ing. Either way, this attack was not orchestrated from the insecure IoT devices that many fear will continue to grow massive botnets in the coming years.

Locations

There is also a fair distribution of locations for those top ten hosts - France, Netherlands, Germany, Indonesia, the United States and Vietnam are all represented. So where did all of the devices come from?

For a nicer view, open a full screen view of the map above.

This fuller picture reveals that in addition to the high concentration of requests from OVH servers in France, there were also a lot of compromised devices in certain American cities. Who owns the networks these devices sit on? This time around I used InfobyIP's Bulk Lookup Tool, which kindly does not limit the number of lookups per day (or at least I didn't hit the limit):

In Scottsdale, Arizona, all devices are on GoDaddy.com, LLC (GoDaddy) networks.
In Brea, California, all devices are on New Dream Network, LLC (Dreamhost) networks.
In Provo, Utah, all devices are on Unified Layer (United Layer)* networks.
In Fort Lauderdale, Florida, all devices are on InternetNamesForBusiness.com (INFB) networks.

* The United Layer devices appear to be a small mix of regional web hosting services, with the only major outlier being one device on HostGator.

The total requests in these cities were not nearly as high as what came from OVH, but they helped the total number of devices from the United States more than double those from France. These network owners also further advance the theory that the devices were compromised website. It is possible that they were not able to fire off any many requests as OVH's servers because they are smaller website that hit resource caps.

A lot more digging could be done here to identify and report individual compromised websites, but I suspect that would be an adventure of it's own. And one worth taking if time ever allows...

User Agents

All of the devices used in the attack had the same user agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0). User agents are very easy to fake, so it's difficult to read much in this. I had not seen the Trident token in user agent strings before and found that it apparently has to do with Internet Explorer's compatibility mode. Perhaps this indicates that the compromised machines used in the attack are servery outdated Windows virtual machines, but there is no definitive way to make that connection.

One other interesting "user agent" string in the logs is () { foo;};echo; /bin/bash -c \"expr 299663299665 / 3; echo 333:; uname -a; echo 333:; id;\". This string, along with requests to some known exploitable entry points, attempts to execute a command to reveal some potentially useful information for compromising the server. If it had succeeded, the attacker would have received the detailed name of the operating system (uname -a) and user and group information for the user running the web server (id).

The part that I was unsure about is the expr 299663299665 command, which is apparently related to the Shellshock vulnerability. The command itself and the result, 99887766555, turn up a number of results on the web but nothing that seems to connect the dots on between the command and the vulnerability.

I suspect that this is an automated worm infection attempt meant to compromise the host being DDoS'd and increase the power of the botnet.

OOM Killer

Ultimately, the DDoS worked like a charm. It only took about 20 minutes to knock out the database server, completely crippling the Drupal-based website. The burst of 4,000+ requests in one minute caused enough hits against the database to trigger the dreaded oom killer, shutting the service down for good (until I brought it back up manually, anyway).

I have run in to oom killer in other low memory VPS situations in the past and it certainly is not a fun issue to iron out. Luckily, this instance was entirely related to the DDoS attack and not some deeper, more obscure memory issue. Once the attack stopped, things were able to return to normal pretty easily.

In the end, I examined some other logs and decided to restore the machine from a previous image just in case. If the site becomes a frequent target of this sort of attack I will evaluate mitigation options, but I suspect that this particular attack was simply a random test of a small botnet.

Loading Plotly Graphs on Demand with Waypoints

2016-11-23T15:42:00-05:00

In my last post, 12 Years of Gmail, Part 4: Chat, I included eight Plotly graphs on a single page. All the graphs worked correctly, but the page was taking almost four seconds to render any content at all and up to 6-8 seconds to load completely without cached elements. By contrast, the landing page of chrxs.net takes less than a second to load with visual content rendering almost immediately. The site is intentionally designed to be light weight and uses very few resources on a standard load. But Plotly graphs require a big (1MB+ uncompressed) JavaScript file in order to load with all the bells and whistles. What can be done to improve this slow load time, particularly when many graphs are on a single page?

Film strip before optimization (webpagetest.org)

The page load film strip above shows almost three whole seconds before any content is rendered. The obvious first step was to move the loading of Plotly's large JavaScript file from the page head (which loads before content is rendered) to the end of the page body, theoretically allowing the page's content to be partially loaded and rendered earlier. However, doing this created a bit of problem because the Plotly graphs were loaded in the body, like so:

<div class="overflow-x-container">
    <div id="62c34418-006c-4aaf-8df7-15d5ad556369"></div>
    <script type="text/javascript">
        window.PLOTLYENV=window.PLOTLYENV || {};
        window.PLOTLYENV.BASE_URL="https://plot.ly";
        Plotly.newPlot(
            "62c34418-006c-4aaf-8df7-15d5ad556369",
            [...],
            {...}
        );
    </script>
</div>

This means the Plotly command is needed in the body, so Plotly's JavaScript must be loaded before any of the graphs. There are a number of ways to solve this, but the most appealing one I came across is the Waypoints library. Basically, Waypoints provides a framework for executing actions as a user reaches a certain point on the page. This is a common practice for website with lots of images and/or "endless scrolling" features. To make use of it in this situation, I simply removed the JavaScript from the code excerpt above and added the following below the Plotly JavaScript load:

new Waypoint({
  element: document.getElementById('62c34418-006c-4aaf-8df7-15d5ad556369'),
  handler: function() {
    window.PLOTLYENV=window.PLOTLYENV || {};
    window.PLOTLYENV.BASE_URL="https://plot.ly";
    Plotly.newPlot(
        "62c34418-006c-4aaf-8df7-15d5ad556369",
        [...],
        {...}
    );
    this.destroy();
  },
  offset: '100%'
});

Line 1 identifies the element ID that will trigger the handler action in Line 2. This ID is the container for the graph that the handler will load. The only addition is Line 10, which destroys the waypoint after it runs once in order to prevent the Plotly graph from loading itself over and over again. This causes the graph to only load a once when the user scrolls to the point on the page that should display the graph.

After I replaced all eight graphs with Waypoints, the page load performance improved massively:

Film strip after optimization (webpagetest.org)

With these changes the page takes less than one second to render the first bit of content - this can be applied to any number of Plotly graphs on a single page and brings the page load time much more in line with the rest of the website (which does not include the Plotly Javascript at all).

12 Years of Gmail, Part 4: Chat

2016-11-18T20:00:00-05:00

This post is part of my series, 12 Years of Gmail, taking a look at the data Google has accumulated on me over the past 12 years of using various Google services and documenting the learning experience developing an open source Python project (Takeout Inspector) to analyze that data.

With the Finishing Touches in place, it's finally time to start looking at some of the data in my Google Takeout Mail export file. What better to start with than the Google Talk (or Google Chat, as I will refer to it) content stored within!

I am starting with Chat because I was surprised to find it all stored in the export file. It makes sense as chat history is accessible from the Chats link in the old Gmail interface (I couldn't find an equivalent in Inbox). My surprise led to curiosity and my curiosity led to obsession with trying to figure how the Chat data is stored and what information each messages contains. It turns out there are quite a few things that can be gleaned from these chat messages -

Chat vs. Email Usage

This is the first graph I hacked away at and luckily it provides a neat little anomaly to look at. Undoubtedly the first thing most people will notice looking at this graph is the big spike in Chat activity that occurs in 2012. Zooming in on this (click and drag in the graph) reveals that the spike occurred in October 2012. One other event that may be noticeable (if much less so) is the dip in activity around May to August 2009. What is the relevance of these dates? Two very important events in my life:

On 12 June 2009, my plane touched down in Burkina Faso and I began my initial two years of service in Peace Corps Burkina Faso. Internet service in Burkina was rare and unreliable. Most of the time we had to use the Basic HTML version of Gmail without Chat support.
- The big dip in August 2010 was the result of a prolonged outage of the Internet service in the town I lived in at the time (Yako).
- The jump in activity in September & October 2011 represents my one trip back home before returning to Burkina for a third year.
On 10 October 2012, after returning from a little over three years of service in Burkina, I moved to Washington, D.C. to start a new job. The huge jump in Chat activity is a result of the fact that I was separated from my future wife before she was able to move to D.C. in late October.

According to Wikipedia, Google Talk was first introduced in August 2005. Apparently I did not start using it until February 2006. But it certainly took over my inbox...

Chat vs. Email Usage (Cumulative)

In a mere three months (by May 2006), the number of Chat messages overtook more than three years of email messages stored in my Gmail account. From there Chat continued to grow at a much faster pace than email (outside of the relative flat line while I was in Burkina). This is not particularly surprising given the nature of instant messaging and email, but the volume of Chat messages over emails is much, much higher than I thought it would be.

I only have one small aside regarding the code here - I wanted to build these graphs using a single query and sqlite made that simple with the strftime function and CASE expression (which for some reason does not have an anchor in the sqlite documentation). This query is what is used for both graphs:

SELECT strftime('%Y-%m', `date`) as period,
COUNT(
    CASE WHEN gmail_labels LIKE '%Chat%' THEN 1 ELSE NULL END
) AS chat_messages,
COUNT(
    CASE WHEN gmail_labels NOT LIKE '%Chat%' THEN 1 ELSE NULL END
) AS email_messages
FROM messages
GROUP BY period
ORDER BY period ASC;

Top 10 Chatters

This graph did not turn out particularly interesting for me given the exponential difference between Chat and email messages. It does clearly illustrate that I chatted with my wife (Tony Taylor) a lot. The roughly 18k Chat messages is about 20% of the total Chat messages in my mail file!

Sizing was a bit of a challenge with this graph. As with my previous efforts, the code determines the longest address (by character count) in the result set and attempts to adjust the graph to compensate, like so:

layout = self._default_layout_options()
layout['barmode'] = 'grouped'
layout['height'] = longest_address * 15
layout['margin']['b'] = longest_address * self.config.getfloat('font', 'size') / 2
layout['margin'] = go.Margin(**layout['margin'])

Once again, I guess-and-checked my way to some seemingly random numbers that appear to have the desired effect regardless of font size. Line 2 sets the overall height for the graph and Line 3 adds a bottom margin to (theoretically) fit all the addresses. Previously I did not change the overall size of the graph and this can still result in squished graphs on smaller screens.

Chat Clients

This was a fun one to do because it is a sort of non-obvious way to look at the metadata in each Chat message. I learned about the Resourcepart used by XMPP while I was sorting out how to parse email addresses. At the time my only goal was to get rid of it in order to clean up the email addresses. The data that ultimately ends up in sqlite does not include the Resourcepart except in the headers table, which holds row-by-row records of every raw header from all messages.

Because the standard is apparently not well followed by instant messaging clients, the aggregate information here is really just a best guess. I knew of most of the clients, but also found some others through an old blog post from Google, Third Party Client Use of the Google Talk Service, covering their own investigation of third party clients using the (relatively newly launched) service. I'm sure some are still missing.

As for my own chart, I was a big fan of Adium back when I used instant messaging much more frequently - that's about the only revelation here.

Chat Times

This is a simple view of the hours of the day when I do the most chatting. The graph times are UTC and for the majority of the messages I was likely in the Central or Eastern timezone (UTC−6) or (UTC-5) so it appears that I did most of my chatting in the morning hours.

The one thing I remember about working on this graph is this long line of code for calculating the percentages:

for hour in data.keys():
    percentages[hour] = str(round(float(data[hour])/float(total_messages) * 100, 2)) + '%'

I suspect there is a more elegant way to arrive at the same result so I will have to revisit that graph method in the future. Having to deal with types when combining variables is not an issue in PHP and it is taking me a while to wrap my brain around this in Python. I encounter lots of TypeError exceptions!

print 10 + '%'
TypeError: unsupported operand type(s) for +: 'int' and 'str'

Chat Days

This graph turned out just about as I expected it - I tended to use Chat as a means of communication with friends while at work during the weekdays (so this comports with the Chat Times graph above (and perhaps there is some combined view to consider here...)). Chat conversation dominated about 80% of my communications during the week, but dropped closer to 70% on the weekends as I would generally have more time to compose emails. I suspect that my earlier habits also influence this relationship - nowadays I am more inclined to spend weekends doing other things.

This is another place where I feel there could be a lot done to clean up the code:

chat_percentages = OrderedDict()
chat_messages = OrderedDict()
email_percentages = OrderedDict()
email_messages = OrderedDict()
for row in c.fetchall():
    dow = calendar.day_name[int(row[0]) - 1]  # sqlite strftime() uses 0 = SUNDAY.
    chat_percentages[dow] = str(round(float(row[1]) / sum([row[1], row[2]]) * 100, 2)) + '%'
    email_percentages[dow] = str(round(float(row[2]) / sum([row[1], row[2]]) * 100, 2)) + '%'
    chat_messages[dow] = row[1]
    email_messages[dow] = row[2]

This seems like a lot of work to make pretty simple calculations. I would also like to cut down on the number of dictionaries, if only to make the code read a little bit easier. Currently the up-side is that this setup makes it dead simple to craft the graph building statements.

Chat Durations

This was a particularly interesting graph to work out due to the way that Chat messages are stored in the Mail file. An example message entry for Google Chat content looks like this:

From 1234567890123456789@xxx Sun Oct 04 11:20:57 +0000 2015
X-GM-THRID: 1234567890123456789
X-Gmail-Labels: Chat
MIME-Version: 1.0
From: Bob Thompson <bob-thompson@domain2.tld>
Content-Type: text/html

Hello!

Notice anything missing there? There is no To or Date header! Luckily, the date is available on the From line and its position and format is consistent for all messages - it is always the last 30 characters in the string so string[-30:] captures the date and time easily. But how can this be used to figure our chat durations? In combination with the X-GM-THRID header!

The Thread ID connects multiple messages together in to threads, which helps to string together messages and find things like durations and participants (remember that missing To header?). Takeout Inspector handles these dates during the import process and also brings in each message's Thread ID in the messages table. With these data points, the following query will produce chat durations:

SELECT strftime('%s', MAX(`date`)) - strftime('%s', MIN(`date`)) AS duration
FROM messages
WHERE gmail_labels LIKE '%Chat%'
GROUP BY gmail_thread_id
HAVING duration > 0;

Pretty simple really. Unfortunately, some messages have the exact same time for the entire thread so these end up with a duration of zero. I have not been able to find any common factor that indicates why this happens. Single message threads also come up zero and originally these results were included and categorized as "Unknown". In my case this accounted for almost 20% of all my Chat messages so I elected to add Line 4 above to exclude those results from the set (see commit #4739987).

Chat Thread Sizes

Here again is the very distinct drop in activity lining up with what appears in the Chat vs. Email Usage graph above. I don't think this graph is particularly good (for my mail file) given how clustered up everything is, but it does do a further job of illustrating that down time and it's at least somewhat interesting to see the "waterfall" effect from larger to smaller threads.

This graph is largely just an attempt to get away from the typical set of line, bar, pie etc graphs. The size of each circle represents the number of messages in the Chat thread (bigger circle == more messages). Hover over any individual circle to also see:

Message count
Thread date
Thread participants

All of these data points are retrieved with the following query:

SELECT gmail_thread_id,
strftime('%Y-%m-%d', `date`) AS thread_date,
COUNT(message_key) as thread_size,
GROUP_CONCAT(DISTINCT `from`) AS participants
FROM messages
WHERE gmail_labels LIKE '%Chat%'
GROUP BY gmail_thread_id;

Line 3 is the important bit - it uses the GROUP_CONCAT() function to pull in all the participants that it can find for each thread. Group chats unfortunately use special addresses that identify the group as a whole. These is actually a unique identifer for each participant included in the address, but there is no way to determine who the participant is beyond the random string of characters assigned. Currently this convention is ignored so groups just show up as a single participant in this graph.

Continue on to Part 5: Mail...

12 Years of Gmail, Part 3: Finishing Touches

2016-11-12T20:00:00-05:00

This post is part of my series, 12 Years of Gmail, taking a look at the data Google has accumulated on me over the past 12 years of using various Google services and documenting the learning experience developing an open source Python project (Takeout Inspector) to analyze that data.

After spending last week Bootstrapping things and, somewhat related, working my way around Pelican, today I have tried to tie up loose ends so I can start spending more time thinking about what information I can get from all this data. While the package is far from complete, these "finishing touches" ended up being the three themes of this morning's work -

Implementing a Settings File

While thinking about how to customize graphs (more on that below) and allow for changes to styles without too much effort, it struck me that there is likely some common ("Pythonic") way to handle settings. And, of course, there is - it's called ConfigParser and it's extremely handy.

To get my feet wet, I created a settings.cfg file with the following contents:

;settings.cfg
[mail]
anonymize = False
db_file = data/email.db
mbox_file = data/email.mbox

It now becomes possible to do a mail import with anonymized data without having to provide inline parameters or keyword arguments. While this is not a major item of interest at the moment, my hope is that down the line this convention or some future variation of it will make for a simple way for users without Python experience to take advantage of this project.

Next, a couple lines of code wherever the settings are needed will get them loaded up for easy access:

import ConfigParser
config = ConfigParser.ConfigParser()
config.read(['settings.cfg'])

print config.get('mail', 'anonymize')
False
print config.get('mail', 'db_file')
data/email.db
print config.get('mail', 'mbox_file')
data/email.mbox

Here it is simply necessary to "read" the settings file in to ConfigPasrer and then provide the "section" and setting to ConfigParser.get() in order to retrieve the information from the file. Previously, changing the location of the mbox file would require finding where it is set in the code and changing it. Now, the setting can simply be changed in settings.cfg and put to use throughout the package.

As I began to refactor the code to use ConfigParser, I hit some trouble deciding how to provide default values if a config setting is missing. Luckily, the Python documentation provides a very clear note about just that in the RawConfigParser.read() section:

An application which requires initial values to be loaded from a file should load the required file or files using readfp() before calling read() for any optional files.

This can be put to use by creating another file, settings.default.cfg, with similar contents and adding one line to the code from above:

; settings.default.cfg
[general]
color = blue

[mail]
anonymize = True
db_file = path/to/database.db
mbox_file = path/to/mail.mbox

import ConfigParser
config = ConfigParser.ConfigParser()
config.readfp(open('settings.defaults.cfg'))
config.read(['settings.cfg'])

print config.get('general', 'color')
blue
print config.get('mail', 'anonymize')
False
print config.get('mail', 'db_file')
data/email.db
print config.get('mail', 'mbox_file')
data/email.mbox

Note the change at Line 2 above - this tells ConfigParser to read settings.defaults.cfg before settings.cfg. The output shows how the config values end up:

general.color is "blue" because it is not provided at all in settings.cfg.
mail.anonymize is False because settings.cfg overrides the True setting in settings.defaults.cfg.
mail.db_file and mail.mbox_file are also set by settings.cfg.

Now the code can still rely on config.get('general', 'color') to return something instead of raising an error if the value is not set at all in settings.cfg.

Customising Plotly Graphs

The basic graphs I have produced so far were only using Plotly's default settings. There is a great deal of details in Plotly's Python documentation for all types of graphs, settings and layouts. Because I only have two basic bar graphs at the moment, I focused on how to make things look better without a lot of individual effort for each graph. There are a number of issues with the graphs from last week:

Everything is quite squished because of the large margins.
The text is much larger than it needs to be.
The long email addresses do not fit in the graph's viewable area.
There are no axis labels.
There is no graph title.

All of this made for some pretty poor visual representations. To remedy this, I added some config values to my settings file and created a method to provide some of the default layout options for a bar graph:

; settings.default.cfg
[font]
family = Lucida Console, Monaco, monospace
size = 11

[color]
; Primary color and shades
primary = rgb(63, 81, 181)
primary_light = rgb(197, 202, 233)
primary_dark = rgb(48, 63, 159)
; Secondary color and shades
secondary = rgb(255, 193, 7)
secondary_light = rgb(255, 236, 179)
secondary_dark = rgb(255, 160, 0, 1)
; Text colors
text = rgba(0, 0, 0, 1)
text_light = rgba(0, 0, 0, 0.54)
text_lighter = rgba(0, 0, 0, 0.38)

The default color settings used here come from none other than Google's Material design guidelines. I figured this was appropriate given that this package is working with data exported from Google.

Now, the default layout options for bar graphs can take advantage of these newly added settings:

def _bar_defaults(self):
    """Prepares default data and layout options for bar graphs.
    """
    return (
        dict(
            marker=dict(
                color=self.config.get('color', 'primary_light'),
                line=dict(
                    color=self.config.get('color', 'primary'),
                    width=1,
                ),
            ),
            orientation='h',
        ),
        dict(
            font=dict(
                color=self.config.get('color', 'text'),
                family=self.config.get('font', 'family'),
                size=self.config.get('font', 'size'),
            ),
            margin=dict(
                b=50,
                t=50,
            ),
            xaxis=dict(
                titlefont=dict(
                    color=self.config.get('color', 'text_lighter'),
                )
            ),
            yaxis=dict(
                titlefont=dict(
                    color=self.config.get('color', 'text_lighter'),
                ),
            ),
        )
    )

This method simply returns two dicts, one for each parameter (data and layout) of Plotly's plot() method. These dicts provide the following:

Line 6: The color of the bars.
Lines 7-10: The color and width of the bar outlines.
Line 12: The graph's orientation (I changed this to horizontal to make room for longer email addresses).
Lines 15-19: Default font settings for the entire graph.
Lines 20-23: Margins for the graph container.
Lines 24-33: The font color of the x- and y-axis titles.

For each individual graph it is necessary to provide some additional configuration. Here is a modified exert from the Top Senders graph:

addresses = OrderedDict()
longest_address = 0
for row in c.fetchall():
    addresses[row[0]] = row[1]
    longest_address = max(longest_address, len(row[0]))

(data, layout) = self._bar_defaults()
data['x'] = addresses.values()
data['y'] = addresses.keys()
layout['margin']['l'] = longest_address * self.config.getfloat('font', 'size')/1.55
layout['margin'] = go.Margin(**layout['margin'])
layout['title'] = 'Top Senders'
layout['xaxis']['title'] = 'Emails received from'
layout['yaxis']['title'] = 'Sender address'

py.plot(go.Figure(
    data=[go.Bar(**data)],
    layout=go.Layout(**layout),
))

Line 6 makes use of _bar_defaults() to establish the data and layout template dictionaries. From there a few more details are added and then the information is passed to plot() as keyword arguments.

One of the big issues with the initial graphs is that only a portion of each email address can be seen. To remedy this, Line 9 makes a small calculation based on the length of the longest email address (determined while looping through the query result set) and font size setting. Dividing by 1.55 was merely a guess-and-check process for various font sizes that seemed to produce the best results. I suspect this will not always work well.

Unpacking argument lists was the most important lesson I (re)learned while working my way through this. This happens in Lines 10, 16 & 17. By placing two asterisks (**) before each dictionary object as it is passed to the method, the dictionaries are "unpacked" to a group of keyword arguments for the method using the key:value pairs of the dictionary. Without this, it would have been much more difficult to figure out how to prepare these default graph settings.

Now, to compare the results!

The old Top Recipients graph:

The new Top Recipients graph:

The old Top Senders graph:

The new Top Senders graph:

Generating Random Names

Previously, a simple uuid.uuid4() was used to come up with "anonymized" names and addresses for every email address encountered. While effective enough, this created very long (36 characters) strings that made even longer complete email addresses. Take this address for example:

"f47ac10b-58cc-4372-a567-0e02b2c3d479" <f47ac10b-58cc-4372-a567-0e02b2c3d479@domain.tld>

That is 87 total characters and, while the uniformity is nice, looking at hundreds of those one after another became very daunting. Plus, these long addresses were using up precious space on graphs! With little effort I managed to find the very simple names package. What does it do? It generates random names. That's it.

So I replaced the old _anonymize_address() method with one similar to the following:

import names
def anonymize_address(address, name):
    domain = 'domain1.tld'
    anon_name = names.get_full_name()
    return {
        'address': address,
        'anon_address': anon_name.replace(' ', '-').lower() + '@' + domain,
        'name': name,
        'anon_name': anon_name
    }

Very simply, this function makes up a fake domain, gets a random full name from names and then creates an email address and a name to associate with the provided address and name parameters. The actual method also keeps track of these domains and addresses while parsing the data.

What does all this mean? Instead of getting 87 character random addresses like these:

"f47ac10b-58cc-4372-a567-0e02b2c3d479" <f47ac10b-58cc-4372-a567-0e02b2c3d479@domain.tld>
"b2d351b3-4607-48e4-9e17-e8d7266438af" <b2d351b3-4607-48e4-9e17-e8d7266438af@domain.tld>
"8e35be14-f719-416b-bc16-a31e260f4385" <8e35be14-f719-416b-bc16-a31e260f4385@domain.tld>
"79417a64-dd09-400c-93ec-58cbef0d1e01" <79417a64-dd09-400c-93ec-58cbef0d1e01@domain.tld>
"f38e5aac-3211-45b3-bc24-80b3754ae32c" <f38e5aac-3211-45b3-bc24-80b3754ae32c@domain.tld>

We get addresses like these:

"Stevie Fulton" <stevie-fulton@domain1.tld>
"Anthony Scott" <anthony-scott@domain1.tld>
"Tony Pack" <tony-pack@domain1.tld>
"Charlotte Wilson" <charlotte-wilson@domain1.tld>
"Hugh Jackson" <hugh-jackson@domain1.tld>

The longest address in that group is 49 characters, almost half the uuid4-based addresses. Granted, this is random so we may get something 87 characters or even longer, but the non-uniformity has one other big advantage: referencing. If I want to produce and talk about some graph based on email addresses, it will be much easier refer to a name like "Stevie Fulton" than "f47ac10b-58cc-4372-a567-0e02b2c3d479".

Continue on to Part 4: Chat...

12 Years of Gmail, Part 2: Bootstrapping

2016-11-08T16:45:00-04:00

This post is part of my series, 12 Years of Gmail, taking a look at the data Google has accumulated on me over the past 12 years of using various Google services and documenting the learning experience developing an open source Python project (Takeout Inspector) to analyze that data.

Jumping back in to Python has been just as fun as my first experiences with it. After brushing off some of the dust, I have managed to put together a (very) small package that does a couple of basic things with a Google Takeout Mail (mbox) file:

Parsing Email Addresses

The mailbox Python module makes it very simple to get an mbox file in to Python and play around using the mailbox.Mailbox and email.Message classes. Here is an example using my mbox file:

import mailbox
email = mailbox.mbox('/path/to/email.mbox')

# The number of emails in the mbox file.
print len(email.keys())
114407

# The "Delivered-To" header of the first email.
print email[0].get('Delivered-To')
my.address@hmail.com

After using mailbox to establish a database and doing an initial import, I quickly noticed that many email addresses were either oddly formatted or differently formatted versions of the same address. Also, header fields with lists of address were not all the same format and often repeated addresses or had other issues. These inconsistencies made it difficult to aggregate information for specific addresses or otherwise organize SQL queries around them.

Here is an example header from the very first email in my mbox file (all of the actual email addresses are replaced with fake ones):

John Smith <john@smith.com>,
        Joe Wilson <Joe.wilson@hmail.com>, <bobl1943@wahoo.com>,
        "Bill White" <bwhite@frontier.com>,
        Joe Johnson <jj@jandj.com>,
        Mike Proud <myfirstemail@wahoo.com>,
        Mom <Jill-Johnson@hmail.com>, Dad <bob-johnson@hmail.com>,
        "John Irving" <JOHN@johnirving.com>, Big bro <rob.m.johnson@hmail.com>,
        Jane Stevens <jane@stevens-inc.net>, Jane Stevens <jane@stevens-inc.net>,
        Michael Whitehead <mwh823@purdue.edu>

There are a lot of problems to note here:

Addresses are inconsistently separated by carriage returns.
Most addresses are prefixed by a tab character.
Some lines have one address, others have more than one.
One email address does not have an associated name.
Some names are surrounded in double quotes.
One email address is repeated.
Letter case is inconsistent.

None of this is particularly difficult to deal with in Python. Each address _is_ separated by a comma, for example, so creating a list from this header would be simple enough (but the comma is not always guaranteed!). I worked on parsing these fields and addresses manually a bit before finding the wonderful email.utils module.

The method email.utils.getaddresses can convert the above header to a list of name-email tuples without having to worry about the extraneous carriage returns, spaces or other formatting differences in the header:

import email
header = ['''John Smith <john@smith.com>,
        Joe Wilson <Joe.wilson@hmail.com>, <bobl1943@wahoo.com>,
        "Bill White" <bwhite@frontier.com>,
        Joe Johnson <jj@jandj.com>,
        Mike Proud <myfirstemail@wahoo.com>,
        Mom <Jill-Johnson@hmail.com>, Dad <bob-johnson@hmail.com>,
        "John Irving" <JOHN@johnirving.com>, Big bro <rob.m.johnson@hmail.com>,
        Jane Stevens <jane@stevens-inc.net>, Jane Stevens <jane@stevens-inc.net>,
        Michael Whitehead <mwh823@purdue.edu>''']
addresses = email.utils.getaddresses(header)
print addresses
[
    ('John Smith', 'john@smith.com'),
    ('Joe Wilson', 'Joe.wilson@hmail.com'),
    ('', 'bobl1943@wahoo.com'),
    ('Bill White', 'bwhite@frontier.com'),
    ('Joe Johnson', 'jj@jandj.com'),
    ('Mike Proud', 'myfirstemail@wahoo.com'),
    ('Mom', 'Jill-Johnson@hmail.com'),
    ('Dad', 'bob-johnson@hmail.com'),
    ('John Irving', 'JOHN@johnirving.com'),
    ('Big bro', 'rob.m.johnson@hmail.com'),
    ('Jane Stevens', 'jane@stevens-inc.net'),
    ('Jane Stevens', 'jane@stevens-inc.net'),
    ('Michael Whitehead', 'mwh823@purdue.edu')
]

Now some basic Python can clean things up a bit (using the addresses list created above):

addresses = list(set(addresses))
for idx, address in enumerate(addresses):
    [local_part, domain] = address[1].split('@', 1)
    domain = domain.split('/', 1)[0].lower()
    local_part = local_part.replace('.', '').lower()
    address_formatted = local_part + '@' + domain
    addresses[idx] = [address[0], address_formatted]

The above code block removes duplicates (Line 0), breaks each address in to its local and domain parts (Line 2), removes any resourcepart (Gmail stores chat data in the mbox file) (Line 3), converts all strings to lower case (Lines 3-4) and recombines the parts to update the entry in addresses (Lines 5-6).

Line 4 also removes any periods from the localpart. I initially did this because Gmail addresses ignore periods, however it now strikes me that this is likely not universally true! I will have to fix this in a forthcoming push of takeout inspector.

All together these operations create a list of unique tuples with standard relatively email address formatting:

print addresses
[
    ['Dad', 'bob-johnson@hmail.com'],
    ['Mike Proud', 'myfirstemail@wahoo.com'],
    ['Joe Johnson', 'jj@jandj.com'],
    ['Jane Stevens', 'jane@stevens-inc.net'],
    ['', 'bobl1943@wahoo.com'],
    ['Joe Wilson', 'joewilson@hmail.com'],
    ['Mom', 'jill-johnson@hmail.com'],
    ['Bill White', 'bwhite@frontier.com'],
    ['Big bro', 'robmjohnson@hmail.com'],
    ['John Irving', 'john@johnirving.com'],
    ['John Smith', 'john@smith.com'],
    ['Michael Whitehead', 'mwh823@purdue.edu']
]

Importing in to SQLite

Now that the To and CC headers can be parsed in to simple lists, the key data needs to be stored in a way that is easier (and quicker) to work with than Mailbox - enter SQLite. My experience is largely with MySQL so there was a bit of fumbling around getting this set up with Python's sqlite3 module. After I got the hang of it, I started poking through my entire mbox file (3.5GB) to figure out what information to import (for now).

I decided to start with three tables -

messages to store some key information about all emails;
headers to store raw headers for all emails;
recipients to store one-row-per-address entries for any email address in the "To" or "CC" header.

The messages table includes the headers: From, To, Subject, Date, X-GM-THRID and X-Gmail-Labels. X-GM-THRID is a Gmail-specific numeric ID that relates messages in threads and X-Gmail-Labels stores a comma-separated list of labels for each email. Common labels include "Chat" for chat messages, "Sent" for sent emails, Important, Spam, etc.

The Date field is an interesting one to deal with because it needs to converted to a format that SQLite can use efficiently. SQLite relies on ISO 8601 while email Date headers use a format outlined in RFC 2822. Also, Gmail puts date information for its chat messages in a different header that is accessible using mailbox.mboxMessage.getfrom(). Once again using my mbox file as an example:

import mailbox
email = mailbox.mbox('/path/to/email.mbox')

# A typical Date header.
print email[0].get('Date')
Mon, 5 Oct 2015 13:18:10 -0300

# A missing Date header for Gmail chats.
print email[3].get('Date')
None
print email[3].get_from()
xxx@xxx Mon Oct 05 00:37:04 +0000 2015

Both of these possibilies can be handled and converted to ISO 8601 format UTC dates using the email.utils and datetime Python modules. The following function assumes two parameters, the result of get('Date') and the result of get_from():

import email
from datetime import datetime

def get_message_date(get_date, get_from):
    if get_date:
        mail_date = get_date
    else:
        mail_date = get_from.strip()[-30:]

    datetime_tuple = email.utils.parsedate_tz(mail_date)
    if datetime_tuple:
        unix_time = email.utils.mktime_tz(datetime_tuple)
        mail_date_iso8601 = datetime.utcfromtimestamp(unix_time).isoformat(' ')
    else:
        mail_date_iso8601 = ''

    return mail_date_iso8601

Lines 4-7 determine which of the two headers to use.
Line 9 creates a tuple of the date including timezone information.
Lines 10-14 attempt to convert the date tuples to ISO 8601 format.

Using the sane two example emails from my mbox file, the function above produces the following outputs:

print get_message_date('Mon, 5 Oct 2015 13:18:10 -0300', None)
2015-10-05 16:18:10
print get_message_date(None, 'xxx@xxx Mon Oct 05 00:37:04 +0000 2015')
2015-10-05 00:37:04

Associating these formatted dates with the rows in messages allows for queries to be run with date/time parameters. For example, to see how many emails from my mbox were handled in December 2014, I can run:

SELECT COUNT(`date`) FROM `messages`
WHERE `date` >= '2014-12-01' AND `date` <= '2014-12-31';
1086

If this data were not in a database, it would be necessary to loop through all the messages in my mbox to figure this out. Instead, this SQL query takes a measly 31ms.

The remaining tables (recipients and headers) don't deal with anything as "complex" (by comparision) as the messages.date column. Check out commit b6f5de9 on GitHub to see how these tables are populated as of this writing.

(Very) Basic Graphing using Plotly

I have not yet spent a lot of time considering the many Python graphing packages that are available. Overview of Python Visualization Tools is a quick review of some of the more popular packages that exist. After reading that article, I elected to give Plotly a try as it appeared to have the quickest path to interactive graphs (and it did prove to be quite fast). There really isn't much happening on this front yet. I only pushed out this bit of code to get a feel for pulling data from SQLite in to Plotly.

Top 20 Recipients

To produce a Top 20 Recipients bar graph with the default Plotly formatting, I can execute the following SQL from Python and massage the results a bit to create a graph in just a few lines of code:

import plotly.offline as py
import plotly.graph_objs as go
import sqlite3
from collections import OrderedDict

conn = sqlite3.connect('/path/to/email.db')
c = self.conn.cursor()

c.execute('''SELECT address, COUNT(r.message_key) AS message_count
    FROM recipients AS r
    LEFT JOIN messages AS m ON(m.message_key = r.message_key)
    WHERE m.gmail_labels LIKE '%Sent%'
    GROUP BY address
    ORDER BY message_count DESC
    LIMIT ?''', (20,))

addresses = OrderedDict()
for row in c.fetchall():
    addresses[row[0]] = row[1]

py.plot([go.Bar(
    x=addresses.keys(),
    y=addresses.values())
])

One thing that I discovered while working on this code: a typical Python dict is unordered. Luckily, there is a collections.OrderedDict object that respects the order of the SQL query results (Lines 16-18). With that data in a dict, all Plotly needs is the the X and Y lists (Lines 20-23) to produce a simple, interactive graph! And a good deal of Javascript, of course...

Note that the resulting graph uses anonymized email addresses. This is handled by the code, but not something I will be discussing quite yet as I want to look in to some other ways to do this. The method used for these graphs is available to view in commit b6f5de9.

Top 20 Senders

Similarly, a Top 20 Senders bar graph is simple to create from the messages table using the same code as above and this SQL query:

SELECT `from`, COUNT(message_key) AS message_count
FROM messages
WHERE gmail_labels NOT LIKE '%Sent%'
    AND gmail_labels NOT LIKE '%Chat%'
GROUP BY `from`
ORDER BY message_count DESC
LIMIT 20

Continue on to Part 3: Finishing Touches...

12 Years of Gmail, Part 1: Google Takeout

2016-10-28T20:00:00-04:00

This post is part of my series, 12 Years of Gmail, taking a look at the data Google has accumulated on me over the past 12 years of using various Google services and documenting the learning experience developing an open source Python project (Takeout Inspector) to analyze that data.

I have been slowly migrating off of a Gmail email address for a couple of months now - I established this domain, selected an email provider, set up SPF, DMARC, etc. and finally created myself a new email address. I updated the address in all of the obvious places, but still found myself using Gmail frequently to keep up. At some point I realized that the only way to finish the migration would be to do something with all the email I had hoarded away in Gmail.

When I made the transition to Gmail (from a mail server in my basement) back in 2004, I found some tool that pulled all my existing email in to Gmail using POP. So, I thought to myself in 2016, I'll just do that again! I fired up Thunderbird, set up Gmail POP access and started downloading. At some point, thousands of emails in, I decided to check just how many emails I had in Gmail - a little over 30,000. Instead of downloading all of that, I went hunting through the emails and deleted stuff I really didn't need (newsletters, test emails, spam, etc.). It was a nice little trip down memory lane! I managed to cut the total in half, but still wanted a better way to get it all downloaded.

Enter Google Takeout. When signed in to Google, this service allows a plethora of data to be exported from an account - mail, contacts, Chrome settings, Hangouts chat history, location history, etc. I ticked the Mail option and was told I would receive an email when my archive was ready for download.

Google Takeout data selection screen.

A few hours later I received the email alert, downloaded the zipped file and uncompressed it to find a 3.5GB mbox mail file. I could import this in to something like Thunderbird (or any other mail client) and have it to refer to, or find some way to import to my new email provider.

However, what ultimately interested me is just how much data is in this little file - 12 years of my communications on the Internet.

I started the account back when I was 20 years old and imported email going back even further in to my teens!

There could be some interesting aggregate information in all this email, such as which emails and domains I communicate with most frequently, when I send and receive emails, what are the most common words I use in communication, etc. etc. I spent a bit of time poking around GitHub looking for projects that may already do this sort of thing, but didn't find anything of particular interest. So I decided this would be a great excuse to dip my toes back in to Python!

I worked through Learn Python the Hard Way a year or so ago and found that I really enjoyed the language. The code structure and syntax is much less cluttered than PHP (where most of my experience is), there is great documentation, lots of easy-to-use packages and it just seemed more fun to write. This experience felt much more like my earlier years learning about HTML and PHP development on DevShed. Ultimately, I didn't have a good project or time to devote to keeping it up so most of that knowledge has slipped away.

As I began working my way around the Mail export, I realized that other elements of Google Takeout - Chrome data, Hangouts and Location History, for example - may also be very interesting to poke around in (and I also greatly regretted my decision to delete half of my email before doing the Mail export!).

Takeout Inspector will be where I capture the code I work on to this end and hopefully I can come up with some fairly interesting information and insights along the way to capture in this series.