Use Git history to suggest related tests

2019-06-20

You've started a new job (congrats!). For your first task, your PM wants you to change the default behavior of help-desk links to open in a new tab.

This is the sort of task that is either trivial or a trip down the rathole of fragile tests that depended on the original behavior.

As apps grow, two things often happen that make changes like this one slower for developers:

  1. It becomes less obvious which tests might be impacted by a change.
  2. The runtime of the test suite grows such that running the entire suite locally isn't palatable.

Many devs will run any seemingly relevant unit tests, any obvious integration tests, and then let CI tell them what they missed. But when CI takes minutes or tens of minutes to run, the feedback loop grows and this once seemingly simple tweak can derail your morning.

Fortunately, there's an easy way to find tests likely impacted by your change...

Git to the rescue (again)

If you're using atomic commits with Git, you have a rich history that groups files with their related tests.

There's no obvious relationship between a file named link-helper.js and your "Subscription Refund Integration Test" but if the two were changed in the same commit, that's a good hint that they might be related.

So if you make your change in link-helper.js, how can you use Git history to suggest related tests?

The naive version looks something like this

#!/usr/bin/env bash

file=$1
pattern=$2

candidates=$(
    # find commits where the file was changed
    git log --format='%H' -- $1 |
    # show file names from those commits
    xargs git show --pretty="" --name-only |
    # filter to only the provided pattern
    grep $pattern |
    # remove duplicates
    uniq
)

echo $candidates

Save that as suggest-tests somewhere in $PATH and chmod +x it.

Now you can invoke suggest-tests app/js/link-helper.js test/ and see all files with "test/" in their path that changed when app/js/link-helper.js also changed.

A more robust solution

There's a few places that the naive solution isn't ideal:

  1. It won't follow file renames.
  2. It returns file paths that have since been deleted.
  3. It would be nice if the uniq preserved history order (most recently edited to least recently edited).
  4. It would also be nice if you could set a default pattern to avoid specifying it every time.

After some thinking, googling, and false starts, here's the version I'm using today:

#!/usr/bin/env bash

function usage {
    script=$(basename $0)

    echo "$script - use Git history to suggest tests that could be relevant to the provided file"
    echo
    echo "Usage: $script file test_pattern"
    echo
    echo "       Note: test_pattern is optional if \$DEFAULT_SUGGEST_TESTS_PATTERN is set"

    if [ -z $DEFAULT_SUGGEST_TESTS_PATTERN ]; then
        echo "       (\$DEFAULT_SUGGEST_TESTS_PATTERN is unset or empty)"
    else
        echo "       (\$DEFAULT_SUGGEST_TESTS_PATTERN is set to $DEFAULT_SUGGEST_TESTS_PATTERN)"
    fi

    echo

    echo "Example:"
    echo "       $ suggest-tests some_file_name.rb _test.rb"

    echo
    echo "You might want to pipe the results into your test runner with xargs:"
    echo "       $ suggest-tests some_file_name.rb _test.rb | xargs rake test"
    exit 1
}

if [ "$#" -gt 2 ] || [ "$#" -eq 0 ] || [ $1 == "--help" ]; then
    usage
fi

file=$1
pattern=${2:-$DEFAULT_SUGGEST_TESTS_PATTERN}

if [ -z $pattern ]; then
    usage
fi

candidates=$(
    # find commits where the file was changed (following renames on the file)
    git log --follow --format='%H' -- $file |
    # show file names from those commits
    xargs git show --pretty="" --name-only |
    # get the test files from those file names
    grep $pattern |
    # uniqify the names but preserve history order
    awk '!x[$0]++'
)

# get the root in case we're called from elsewhere
git_root=$(git rev-parse --show-toplevel)

# only return candidates that still exist on disk
for candidate in $candidates
do
    if [ -f "$git_root/$candidate" ]; then
        echo $candidate
    fi
done

This solves all our issues and adds some helpful usage instructions. Also, how great is that awk trick?

Running the relevant tests

I live in Ruby + minitest world most of the time so here's an example of how I run relevant tests: suggest-tests some_file_name.rb _test.rb | xargs rake test

Vim integration with fzf

When editing a file, it can sometimes be useful to edit related test files. Here's an example Vim mapping to quickly jump to these files with fzf.

nnoremap <silent> <Leader>S :call fzf#run({
\   'source':  'suggest-tests ' . bufname('%'),
\   'sink':    'e',
\   'options': '--multi --reverse',
\   'down':    15
\ })<CR>

That uses $DEFAULT_SUGGEST_TESTS_PATTERN (which I've set locally to '^test.*_test\.rb$') but you could make a binding for various patterns as you wish.

Closing thoughts

This approach isn't perfect (since you might break a test that shares no Git history with your changed file), and CI will still catch anything you miss. This script has saved me numerous CI feedback cycles over the past year and I hope it does the same for you.


No changelog, no problem

2019-05-20

"Happy Monday!" Your buddy over in Business Intelligence has a question. Some time on November 23rd, a specific tracked event dropped in frequency by half. Fortunately this isn't a mission-critical metric or you would have heard about it awhile ago. Still, what happened on November 23rd?

If you keep an official changelog, maybe your BI buddy can answer this on their own. Otherwise you're going to have to dig into your ol' reliable git log for help.

I've done this spelunking enough times now to write up a little script named what-shipped-on. Since we use feature-branches and merge into a release branch on deploy, we can run the script from the deployed branch to see what commits were merged on a date.

Example:

$ what-shipped-on 2019-05-20
commit 41abc0bd3960b47daadaf7fa1a8ee5bf68d38609
Author: Jeffrey Chupp <jeff@semanticart.com>
Date:   Mon May 20 11:44:06 2019 -0400

    Unify css files

commit c061f869e7ad1d6362d4a8d3836e513ca2a72e59
Author: Jeffrey Chupp <jeff@semanticart.com>
Date:   Mon May 20 11:07:12 2019 -0400

    WebP for images

Here's the script:

#!/usr/bin/env bash

if [[ $# -eq 0 ]] ; then
    echo 'Please provide a date in YYYY-MM-DD format.'
    exit 1
fi

desired_date=$1

# remove date option so remaining options can be passed transparently to git
shift

day_before=$(date -j -v-1d -f "%Y-%m-%d" "$desired_date" "+%Y-%m-%d")
day_after=$(date -j -v+1d -f "%Y-%m-%d" "$desired_date" "+%Y-%m-%d")

git log --after="$day_before" --before="$day_after" $@

The script is pretty straightforward but I'll draw special attention to the use of shift. This allows us to pass additional options to our command that get passed along to git log. So if you want the patch view with short stats, you're in luck: what-shipped-on 2018-11-23 -p --shortstat

May your answers always be clear and your fixes trivial.


Building a Board Game for My Daughter

2019-01-29

I made a Paw Patrol-themed board game for my daughter. It uses both physical and digital components. I wanted to create a game that would bring her delight without bringing me suffering (like so many board games for young children). It didn't end up being a great game but it was enough fun to build and play with Kate that I suggest you try making your own.

I'll talk about the background, the design/implementation, and what worked and didn't work. I'll also use the word "pup" more times than anyone is comfortable with.

Background

My daughter, Kate, is 4 and a half and we describe her as "spirited." When I finish work for the day, she's my other boss. She has a huge imagination and we have a lot of fun together.

One board game we play together is Candy Land. The beauty of the game is in its simplicity: You race to the end of the board by drawing cards and moving to next colored space. If you get a double color, you move twice. Simple to learn and play.

I don't enjoy playing Candy Land. Because of the simplicity, it isn't really fun to play as an adult -- but that's fine since I'm playing for Kate, not myself. For me, the real annoying bit of Candy Land is a mechanic I omitted from the last paragraph: if you draw a card with a sweet treat on it (e.g. the candy hearts), you move to the treat. That's great if the treat is ahead of you, but often times you've passed it long ago. I don't want to think about how many hours of my life I've spent re-playing the same game of Candy Land because Kate and I each drew a card to send us back to a treat near the beginning of the board.

At least it isn't Chutes and Ladders, right?

The other issue I have with Candy Land is one of motivation. In our version, the characters are red, green, blue, and yellow gingerbread men. They're identical besides their color. They lack personality. The game has no sense of urgency beyond wanting to win. You don't really get invested in it.

Kate plays Candy Land but she doesn't love it. It is just another thing for us to do together. She'd rather pretend.


Like many children her age, Kate does love Paw Patrol. At her request, I spend many nights pretending to be one of "the pups" with her. Paw Patrol is basically the opposite of Candy Land. The characters have well-defined personalities and abilities. There's a sense of urgency in each adventure. Kate is invested in an episode of Paw Patrol.

Kate has a Paw Patrol Busy Book that includes 6 small pup figures. One day she decided we should use these figures instead of the gingerbread men to play Candy Land. I didn't realize at the time how much it would change the game. Now she wanted Rubble to beat Chase to the end. Along the way, she mentioned different previous adventures the characters had been on. It infused some much-needed personality and purpose.

I decided to try to build a game that pulled all these things together.

Design goals

I jotted down a few bullet points for the design:

  • The objective is to reach the end first.
  • It should be easy to construct from readily-available materials.
  • Reward-only mechanics (no frustrating backtracking).
  • Make use of the characters' personalities.

Physical Construction

I went to the store and purchased 22x14 inch sheets of paper and color-coding round stickers. I put down a winding path of alternating colors on a sheet of paper. That night Kate and I each rolled a six-sided die to move the pups to the end. It wasn't exactly fun yet, but it was already a game.

board overview

I promised Kate I would later draw the Paw Patrol headquarters (The Lookout) at the end of the path. She reminded me about this for several days before I finally sat down and drew a passable Lookout.

board overview

Reward mechanics & Personality

The pups (it kills me to keep typing that, by the way) already have their own personalities, likes, dislikes, and abilities. Some of that would come across in the role-playing of the characters. I decided the reward mechanics should build on this and felt that a card-drawing approach was the best way to make this happen.

At the beginning of a game, the players each choose a pup as their avatar. They also decide upon a color (or two colors) out of four as the bonus color. When a player lands on that color, they draw a card. The card presents a scenario from the show or a homebrew scenario. In each scenario, something happens requiring the pups' help. If your pup matches the list of pups on the card, you get to move some number of bonus spaces (a random number between 1 and 3) as a reward for helping out. If your pup doesn't match, there's no punishment, you just receive no reward. Here's an example card:

example card

You can demo the cards here.

A card will only ever show two pups at most. If more than two pups could solve a problem, two are chosen at random.

My wife asked me if I was going to print out cards, but I never really considered this. It is easy to keep a laptop or iPad nearby when we play. Having the cards be digital (HTML, CSS, and JavaScript) means that I can tweak them endlessly, introduce more randomness, and change rules dynamically in a way that isn't possible in physical media. Digital + physical games have amazing potential, IMHO.

Kate loved the cards. She recognizes scenarios from the show and helped me write some new scenarios and suggested which pups could solve them.

Successes, Failures, and the Future

Kate enjoys playing the game. She is much more invested in it than other board games we play and liked helping me create the scenarios. This game has helped us extend her pretending into more structured role-playing in a game. We had fun getting here. Mission accomplished.

What didn't work as well is that she's still at that age where "fair" isn't always a clear concept. If I get lucky and draw a few bonus cards that match my pup, that can be frustrating for her. She declared at one point in frustration that "Zuma is the best pup." (This is demonstrably untrue since Zuma is woefully underused in the show and one of the least represented pups in the bonus cards. Pro-tip: Chase is OP.)

With kids, the perception of fair is less important than actual fairness.

The game itself isn't terribly fun. But we'll reach for it instead of Candy Land.

Thinking about the future, it might be more fun for her if the card deck app only included cards for the pups that are in play. This would change the mechanic to always be a bonus rather than ever being neutral. It would probably feel more fair to Kate. A middle ground might be to change the random pup selection for multi-pup cards to favor the pups in play.

Overall, this was a fun experiment and I wouldn't change too much besides adding more scenarios, laminating the board, and the aforementioned card tweaks.

It was good quality time with my daughter, too much time reading the Paw Patrol Wiki, and I learned a little CSS along the way. Not bad.

Maybe you should try something similar with your kid.

Parts List

For completeness, here's the parts list and a link to the current version of the cards: