This post is part of the series, Advent of Code 2017, where I work my way through (part of) the 2017 Advent of Code challenges in order to re-learn some of the basics of Python and reflect on some simple concepts and functionality. All my challenge code is available in cdubz/advent-of-code-2017 on GitHub.

Day 6: max(x, key=y)

• Challenge: Day 6: Memory Reallocation
• Solution: advent-of-code-2017/06

Another simple revelation on built-ins from this challenge: the key argument can be used to modify what is evaluated by the max function. This concept is explained in detail as part of the accepted answer to this Stack Overflow post: python max function using 'key' and lambda expression.

For Python dictionaries, this means that the get() method can be used to return the key of the maximum value in a dictionary. This works because the max function sends each of the dict keys to the dict.get() method and evaluates that result instead of the key itself. Without this argument, max will actually evaluate the dictionary's keys, which isn't terribly useful:

1 2 3 4 >>> max(d) 3 # This is the *maximum value of the dictionary's keys*. >>> max(d, key=d.get) 2 # This is the *key of the maximum value in the dictionary*.
Continue reading

This post is part of the series, Advent of Code 2017, where I work my way through (part of) the 2017 Advent of Code challenges in order to re-learn some of the basics of Python and reflect on some simple concepts and functionality. All my challenge code is available in cdubz/advent-of-code-2017 on GitHub.

Day 1: zip()

• Challenge: Day 1: Inverse Captcha
• Solution: advent-of-code-2017/01

This challenge taught me about Python's built-in zip function. The basic goal is to compare values in a list with specific positional relationships to other values (e.g. next to, X steps from, etc.) in the same list. zip assists with this task by combining multiple lists in to tuples. My initial solution used a construct similar to:

1 2 3 4 total = 0 for a, b in zip(digits[::1], digits[1::1]): if a == b: total += a
Continue reading

I taught myself Python a few years ago by following the wonderful Learn Python the Hard Way (LPTHW) series by Zed A. Shaw. Since then, I have spent a decent amount of time in Python largely in the Django framework. Django is a lot of fun to work with because it abstracts away much of the complexities of developing a full-featured web application in Python. In this way, however, it has also led me to forget some of the basic Python that I learned through LPTHW.

In order to recapture some of those early lessons (and maybe learn a few more Python 3 specific ones), I worked through (part of) the 2017 Advent of Code, a 25-day, language agnostic programming challenge series developed by Eric Watsl. I originally thought about using the series to learn a new language, but eventually realized that I still have a long way to go in Python.

This series explores my (often very basic) revelations and lessons learned while completing 20 of the 25 days of challenges (vacation travel cut the series short for me). The code I used for each day is available on GitHub. Also check out the advent-of-code and advent-of-code-2017 GitHub topics to see the many solutions people have developed in various languages.

• Day 1: zip()
• Day 2: itertools.combinations()
• Day 3: Manhattan Distance
• Day 4: set()
• Day 5: Profiling
• Day 6: max(x, key=y)
• Day 7: RuntimeError: dictionary changed size during iteration
• Day 8: Regular Expressions
• Day 9: Lookahead, Lookbehind
• Day 10: functools.reduce()
• Day 11: Navigating a Hexagon Grid
• Day 12: Recursion
• Day 13: Shallow vs. Deep Copy
• Day 14: Hexadecimal to Binary
• Day 15: progressbar2
• Day 16: One Billion Permutations in 0.535 Seconds
• Day 17: collections.deque()
• Day 18: Negative Numbers and str.isdigit()
• Day 19: The Internet is Not a Big Truck
• Day 20: Reading from a File

Continue reading

Back in April, I learned about Timestrap, a self-hostable, Django-based time-tracking project from a post on HackerNews by Isaac Bythewood. As I have been learning Python in the past year or so, I reached out to Isaac and started contributing to the project. After getting familiar with the core application, I turned my attention to testing and eventually found my way to Selenium, a collection of browser automation tools used for frontend testing.

I had never worked with Selenium or other automated testing products, so it struck me as a great opportunity to get my feet wet in something new. After getting things up and running, we quickly learned that the test results were quite inconsistent across development environments - even to a point that occasionally tests would succeed when run individually, but fail with the full test case.

After much trial and error, we have settled on a (mostly) consistent setup for testing with Selenium, Python and SauceLabs. This produces much better results than testing in development environments and crossing fingers during CI. Hopefully this primer will help others facing similar challenges (as we had a lot of trouble finding good material on the subject).

Continue reading

I was recently shopping online for a specific type of lotion (my skin seems to hate all other types). I am somewhat predisposed to avoid Amazon.com because its size and increasing dominance of online shopping concerns me as it moves closer and closer to "shopping" (in the sense that "searching" means Google to most people). However, my biggest reason for avoiding Amazon.com is simpler: it has become incredibly confusing to shop there. Searching for just about anything will yield thousands of results and it takes (me) a lot of effort to determine which one is appropriate. This can be a great thing in a lot of cases, but more often than not it feels more like navigating a minefield of deceitful listings than comparing competing products.

This lengthy post evaluates one example - Aveeno "Daily Moisturizing" lotion. As with most personal care products, when searching online I pull up a couple of websites to compare prices. In this case I pulled up Amazon.com last, searched for "Aveeno" and was impressed to see just what I was looking to compare (the 18 oz., non-SPF version) as the first result. I clicked through and, curious to figure out the price per ounce, looked to the "add to cart" area. What are are my options?

• Sizes select list with 37 sizes.
• Styles (??) select list with three styles.
• "Subscribe & Save" (pre-selected) and "One-time purchase" radio buttons.
• "Qty" select list.
• "Delivery every" select list (2 months pre-selected).
• "Subscribe now" button.
• "Add to List" button.
• Three "Add to Cart" buttons with different prices.

Well, this is going to be complicated...

Continue reading

Yesterday, while reviewing some logs I came across a curious entry in an Apache error log:

[Wed Apr 19 08:51:48.119666 2017] [core:error] [pid 29210] (36)File name
too long: [client 137.226.113.7:40907] AH00036: access to
/YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForR
esearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongReques
tURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALo
okAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurpos
eWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisI
sAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPu
rposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWe
AreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUs
erAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreSca
nningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyL
ongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePlea
seHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingI
tOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTH
XYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann failed
(filesystem path '[...]')

Formatted to plain English: Yes, this is a really long request URL but we are doing it on purpose. We are scanning for research purpose. Please have a look at the user agent. Thanks!

What does the user agent for this request have to say?

Continue reading

On Sunday, 19 March around 10AM I received an email with the subject line IMPORTANT: ProtectMyID Surveillance Alert. I was busy at the time so I flagged the message and moved on with my day. I had received these alerts a few times before when applying for credit cards and getting a home loan. While I hadn't done either of those things lately, I still was not particularly alarmed by the subject alone and had many other things on my mind. Going about the busy day, I promptly forgot about the email.

Before starting the work day the next morning, I noticed and remembered the email in my inbox. Why do I even have ProtectMyId? Because my personal data was part of a breach at some point. Which breach? I don't even recall - there have been so many. I opened the email and logged in to find that a new credit account with a provider I had never used had been opened in my name. Surprise! I am a victim of identity theft.

What do I do now?

Continue reading

Having fallen behind a bit on Takeout Inspector, the 12 Years of Gmail series and some other projects, I decided to try to put something very simple together from beginning to end and actually launch it. One of my previous posts, Examining the Remnants of a Small DDoS Attack introduced me to the Python package ipwhois and the alternative WHOIS system RDAP. This eventually led me to a quick and simple project called RDAP Explorer...

Continue reading

After taking a look at the chat data in my export, I am finally ready to move on to some of the actual mail! Much of what I will look at here is pretty similar to what I was able to turn up with chat data. I tried to branch out a bit, bringing in a new package to create word clouds, and also refactored some of the Takeout Inspector code to form the beginning of a more "formal" report generating process (instead of just spitting out a single HTML file with only a certain subset of the data). Hopefully I can continue to improve this to a point allowing for easier report generation for any user. Anyway, on to the mail data!

• Top 10 Recipients
• Top 10 Senders
• Thread Durations
• Thread Sizes
• Activity by Day of the Week
• Activity by Hour of the Day
• Label Usage
• Subject Word Cloud

Continue reading

On Sunday (27 November 2016) a small website that I advise on was the victim of a DDoS attack that managed to knock the site offline. I received notice on Monday that the website was not working. I was able to ssh to the web server and quickly found that the database service was stopped. After a brief examination of the database logs (nothing too out of the ordinary), I started the service back up and sure enough the website came back online. As the website runs on Drupal, I logged in to take a peak at the Recent log messages and found hundreds of records of log in attempts from a lot of different IP addresses. User accounts on the website are only used by administrators to update content, so it was clear that the site was hit by a DDoS attack!

Continue reading