Solving Advent of Code Puzzles with GitHub Copilot: Day 2
- 4 minsWARNING: Spoilers for Advent of Code below
In my previous post, I tried to use GitHub Copilot to solve some puzzles for Advent of Code this year, in an attempt to learn more about Copilot, and participate in the advent calendar for the first time.
Solving Puzzle One
For the first day of puzzles, I was pretty impressed with Copilot’s ability to give a solution to both puzzles very easily and quickly. I wanted to use Copilot for the second day of puzzles to see what else it can do and if there’s anything else I can learn about it.
So, here’s a video of me attempting the first puzzle:
This time around, it looks like there were a few details that Copilot did not manage to address.
1. Using an Invalid Set of Instructions
The valid set of instructions for the puzzle are forward, up, down, however, Copilot suggested F, U, and D.
2. Incorrect Input Parsing
Another thing that Copilot suggested was f.read().split(','), which resulted in the calculate method raising an error.
Since there weren’t any examples of the input file in the docstring, I think that Copilot was possibly lacking the context it needed to be able to provide a better input parsing method like f.readlines() (I test this theory later).
3. Confusing Up and Down
One last detail that was missed was that moving up actually decreases the depth, and moving down is supposed to increase the depth, so I had to swap those.
Adding More Context
Overall, not too bad! Copilot got me about 90% of the way to a solution the first time around, and I really didn’t have to think much about what I was doing up until I started seeing errors.
I wanted to see if adding more context (more of the description from the first puzzle), would give Copilot the information it needed to get these things right.
After adding more context, as I began to type, I saw that Copilot was suggesting code that looked almost exactly the same as the first attempt.
However, as I typed more and got to typing out def calculate_horizontal_position_and_depth, Copilot then gave a much better suggestion, and I spent the rest of the video actually just trying to fix a bug I introduced in the input file.
Testing Missing Context
At this point I had asked myself: was the more complete problem description the data that resulted in the better suggestion? Or was it simply the result of me continuing to type through the initial suggested code (which would be less exciting)?
So, I went ahead and wrote in only the context from the first attempt as a docstring again, and tried to type past the code suggestion to see if I could get the same result as the second attempt.
And, lo and behold, I could not get Copilot to suggest the same code from the second attempt. I’m not too sure what this means just yet for future puzzles, but I did find an interesting research recitation buried in GitHub Docs about Copilot regarding context.
Solving Puzzle Two
Solving the second puzzle with Copilot was a little more tricky. I had to start searching through the list of synthesized solutions before I could find anything that was useful. In fact, there was even some gibberish / non-sense code suggested, so I’m wondering what was missing from what I typed that affected the suggestions.
Once I had my desired method name typed out, Copilot did produce an actual full solution (though this time around the only thing that Copilot missed was attributing increases / decreases of depth to up / down rather than down / up).
I’ll be interested to see what the next two or three days look like. You can expect another post from me in a few days!
EDIT: So, after I made this post, I actually started a new role at GitHub, and decided to focus my time on onboarding and also getting some rest over the holidays. Sorry! Next year I’ll maybe get back into this.