Practical Career Advice

• The Magic Loop by Ethan Evans
  • Build your career by making a partnership with your manager that accelerates you both. This is the framework I personally use to grow.
  • Longer form video version.
• Don’t call yourself a programmer by Patrick McKenzie
  • Frame your career as solving business problems, not building software. Work in profit centers not cost centers. An oldie but a goodie.
• Three Things Blocking Your Promotion to Staff/Principal Engineer by Steve Huynh
  • Your org, project and manager may not have the scope for you to grow (but they won’t tell you that).

Staff+ Engineering

What does it mean to be a Staff+ Engineer, and how do you get there?

• Staff Engineer by Will Larson
  • Mandatory reading for any Staff+ engineer. A comprehensive overview of what it means to be Staff+, how to get there and the different Staff archetypes. Also available as an audiobook.
• Why and How to be a big-tech Principal Engineer by John Miller
  • Specifically covers Principal and how it differs from Staff.

Leadership Stories

Inspiration from others can be hugely clarifying and motivating.

• Peterman Pod by Ryan Peterman
  • Long form candid conversations with engineering leaders.
  • Highlights:
    • Instagram Principal Engineer on Promotions, Breaking Prod, Tech Leading
      • How to tech lead, manage risk and spend your time.
    • OpenAI & Meta Distinguished Engineer (IC9) On Working With Zuck, Carmack & Career Growth
      • Breakdown of the impact and scope expected from IC1 to IC9.
    • Amazon VP On Stack Ranking & PIPs, Working With Bezos, His Promotions Candid candid dive into how big tech promos and layoffs actually work.
    • Shopify Distinguished Eng (L10) on Principal+ Engineering, Career Story, Regrets
      • Great exploration of Principal+ engineering and the “specialized generalist”.
• Staff Engineer Stories by Will Larson
  • Stories from the Staff Engineer book.
• StaffEng Podcast by David Noël-Romas and Alex Kessinger
  • Interviews with Staff+ engineers across the industry.

Management

As a Staff+ engineer you should understand how your manager thinks, and be good at management-shaped work.

• High Output Management by Andy Grove
  • The canonical engineering management text, a must read.
• An Elegant Puzzle: Systems of Engineering Management by Will Larson
  • The engineering management counterpart to the Staff Engineer book.

Money

Money matters. With some minimal effort you can make a lot more of it.

• Salary Negotiation by Patrick McKenzie
  • The ROI on salary negotiation is absurd. Think in terms of fully loaded costs and the business’ goals.
• The Trimodal Nature of Software Engineering by Gergely Orosz
  • Why does the same role pay 2-4x as much as different companies? How can you earn an American tech salary while working in Europe?

I hope you find these resources as useful as I have. If there’s anything I’ve missed please reach out and let me know!

But the truth is I didn’t even own a Game Boy, yet alone Pokémon Yellow. I prayed against all odds that I’d find a Game Boy on sale for £1 at Poundland, which only went to show that I didn’t have a great grasp of economics.

I never got to play Pokémon Yellow. I did, however, have a Pokémon Yellow in my head. From the Master Guide I had gained a surprisingly deep knowledge of the game without ever actually playing it. In my head you could find Moltres on the second floor of Victory Road, just as it is in the real game (I am assured). I didn’t know what it felt like to catch Moltres, sure, but I knew its stats, where to find it and what it looked like.

The Master Guide was so comprehensive that you could probably reverse-engineer the whole Pokémon Yellow game from it. Stop a minute to think about what that would look like. Really, what would it look like? It wouldn’t be the Pokémon Yellow, but it would be something a lot like Pokémon Yellow. Because of this we will say that the Master Guide is a good model of Pokémon Yellow.

We could do the same reverse-engineering exercise using a game review. Could we recreate a high fidelity Pokémon Yellow game just from a review? I don’t think so. The review would underspecify the game, requiring almost all the details be interpolated. We could create another game that shared similar mechanics, but it wouldn’t be much like Pokémon Yellow. Because of this we will say that a game review is a bad model of Pokémon Yellow.

Game guides are good models of games. In 2023 game guides have come a long way, and are now built collaboratively using wikis like Bulbapedia. A game wiki is an excellent model of a game. Given a game wiki we should be able to recreate the corresponding game a reasonable level of fidelity.

In a normal year this is where the thought experiment would end, but 2024 is not a normal year. With the recent advances in GenAI we now have access to capabilities that were once the reserve of science fiction. Using text as the universal interface we can create text, art, audio and code from raw compute. With this technology we could mechanically create a game from just its model. This ceases to be just a thought experiment: we build a system that takes the Pokémon Yellow wiki in and pushes a Pokémon Yellow game out.

Why is this interesting? Suppose we want a new version of Pokémon Yellow with a new Pokémon creature. We don’t need to change the game’s source code, we just need to edit the wiki and then press the “remake game” button. Modding a game becomes as simple as editing the wiki. Creating a game becomes as simple as writing its wiki.

Despite the GenAI revolution our capabilities aren’t yet advanced enough to do this in the general case. However, I believe that today we are already capable of doing this in a restricted environment. I think we could mechanically create a Game Boy era Pokémon game from a model by using a static game engine and dynamically creating the areas, enemies, items etc. from the wiki. It’s not easy, but I think it’s possible.

This is what I’m hacking on right now: a wiki that makes games. Check out langengine.com to join the waitlist.

Some closing thoughts:

• How do we handle the non-determinism of generating parts of the game that are underspecified?
• What happens when we use AI to write the wiki?
• Does this thinking apply outside games?
• How good of a model is Wikipedia for reality?

GALGA is like Magic: The Gathering in that each card has its own special rules that change the game. These rules can be simple, like “heal 10 life”, or more complex, like “make healing deal damage instead”.

The rules are a Domain-Specific Language (DSL) for describing what happens in the game. They are like a mini programming language that runs only on the machinery of the game. Interestingly (or perhaps frighteningly) rules can rewrite other rules at runtime. The rule “make healing deal damage instead” rewrites healing rules to damage rules. The rules DSL can actually metaprogram itself!

I implemented GALGA in Haskell, and one of the many motivations for this is that Haskell has great support for embedded DSLs. It lets you write a DSL while retaining the full and considerable power of the Haskell type system. It shouldn’t be surprising that a language steered by programming language researchers would be good at writing programming languages 😉

There are a few different ways to use embedded DSLs in Haskell (Free Monads, Tagless Final and more). Free Monads are usually the wrong choice as they are worse than the other options in both performance and type complexity, but I chose to use them anyway for reasons I’ll be digging into below.

A note for the non-Haskeller: the word “monad” might have scared you off there! If you don’t have a vague understanding of monads, I wouldn’t recommend looking for an explanation online. Rather, the only way to truly understand them is to get stuck into Haskell and experience the problems that monads try to solve. There is no other way!

So what is a “free” monad?

Free as in freedom?

Free as in beer?

No, mathematically free.

A free monad is a monad that satisfies the monad laws and nothing more. It has all the stucture of a monad and none of the “effects”. I like to think of it as a program AST (Abstract Syntax Tree) without an interpreter. The free monad represents some computation, but it doesn’t define what that computation actually means until paired with an interpreter. Different interpreters can interpret in different ways.

Other flavours of DSL can swap implementations in much the same way free monads swap out interpreters, so why choose free monads? Their unique edge is that the “AST” is maintained and can be introspected and metaprogrammed. Earlier I mentioned that this is a key requirement of our DSL, because we have cards that rewrite the rules of other cards at runtime.

(side note: “freer monads” improve on the performance and interpreter composability of free monads, but I haven’t played much with them yet. It looks like they are better but there is a lot less beginner-friendly writing about them.)

GALGA implements multiple embedded DSLs using free monads. There is a high level DSL for game rules as discussed, and lower level DSLs for changing state directly and visualization. The rules DSL interprets to the lower level DSLs as needed, usually embedded together.

The primitive DSL is a low level DSL for changing the game state, consisting of only basic getters and setters. When interpreted it mutates the game state. I like to think of it as the “machine code” to which the rules DSL compiles.

-- Example primitive DSL program
do
  let which = PlayerA
  life <- getLife (other which)
  setLife (other which) (life - 4)

The animation DSL is a low level DSL for visualization. When interpreted it calculates state diffs and associated animations to send to players. Because it relies on accessing the game state it is only interpreted when embedded with the primitive DSL.

-- Example anim DSL program
Anim.hurt PlayerB 4 Slash

The rules DSL is the high level DSL for describing the game rules, things like drawing cards or healing. When interpreted it compiles to the primitive and animation DSLs embedded together.

-- Example rules DSL program
do
  let which = PlayerA
  hurt 4 (other which) Slash
  draw which which 1

Because Haskell is a lazy language we only compute the interpretations of the rules DSL that we actually use. In some case we only need the state change, for example when the AI evaluates moves. If we simply don’t use the animations they are never computed.

Because Free Monads give us ASTs for our DSLs, we can metaprogram them by transforming those ASTs. For example, the rule “healing deals damage instead” can be implemented as a rewrite rule that replaces healing AST nodes with damage nodes. This is why we have a separate DSL for rules rather than making it an abstraction over the primitive DSL. It is impossible to identify healing AST nodes when the AST is made of just getters and setters.

-- Example healing to hurting rewrite rule
rewriteRule :: Rules.DSL a -> Rules.DSL a
rewriteRule (Heal l w n) = Hurt l w Curse n
rewriteRule dsl = dsl

Embedded DSLs have proved a powerful pattern. I initially planned to use the DSLs only for card rules, but they have expanded to be used for every part of the core game logic. They are the killer feature of Haskell and I will definitely be using them again in the future. As for free monads, I wouldn’t recommend them for most use cases as they are relatively complex and slow compared to alternatives. However, metaprogramming was crucial for this project and so they were the right choice.

I hope you gleaned something useful from this blog post! If you did, consider checking out GALGA for free in your browser or peruse the source on GitHub 🙇‍♂️

May your wheel keep turning.

Deploying the application involves spinning up a new Digital Ocean droplet using docker-machine. Earlier this week every deployment started mysteriously failing with the following error:

Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded.

It turns out that the latest Docker version doesn’t work with docker-machine. To make things right again you’ll want to pin the Docker engine version like so:

--engine-install-url "https://releases.rancher.com/install-docker/19.03.9.sh"

It took a while of Googling to find the issue so I’m hoping this post can save someone else the headache.

Tests are usually written to check that code is correct. This is a lofty and noble goal, which makes it easy to forget that they can be used in other ways too. I like to use tests to communicate with future developers who are making certain predictable changes, reminding or warning them through the failure message. I call these tests “tripwire” tests.

Example: Making sure temporary hacks are temporary

When working on a legacy project I found a bug in the framework we were using. The bug could be worked around with an ugly hack, but it was easy to fix with a PR so I naturally did so. Unfortunately we were stuck on an earlier framework version and not yet ready to upgrade to the latest fixed version. We were stuck with an ugly temporary hack for now.

It seemed excessive to fork the whole framework, but I wanted to make sure the hack would eventually be removed even if I wasn’t around to do so. I wrote a tripwire test which failed when the framework was updated to the fixed version. The failure message told the updating developer to remove the hack as it was no longer needed, acting as a kind of forced documentation.

I left the project, the library was upgraded, the tripwire test was tripped and the hack was removed. Happy ending!

Example: Highlighting when conditions change

I was iterating fast on an admin tool that used the Wikidata API to make a SPARQL query. Conditions were such that I didn’t need to worry too much about SPARQL injection: the tool was private and the database was read-only. I did some naive validation of parameters and called it a day, but I was worried about what might happen if these conditions changed. For example, what if we moved from the external read-only database to our own internal read/write database? I added some documentation warning about this but it didn’t seem like enough.

I wrote a tripwire test which failed when the value of the API endpoint URL changed, reminding the developer about the security assumptions. Under continuous integration the developer will be forced to acknowledge the warning in a way that just documentation can’t do.

Conclusion

Tests can be a powerful tool for communication. Don’t just think of them as testing correctness, consider testing other assumptions as well. I’d love to hear other ways you’ve used tests to communicate.

I ended up using the raw install script from a previous GitHub release just like I would use the script hosted at https://get.haskellstack.org. The following let me install Stack 1.9.3:

sudo curl -sSL https://raw.githubusercontent.com/commercialhaskell/stack40cf7b37526b86d1676da82167ea8758a854953b/etc/scripts/get-stack.sh | sh &&

The diligent player Catherine32 has been grinding the CPU player for experience points. She’s making good progress but not enough; she wants more, much more. She wonders how much experience CPU must have, playing against her and everyone else with no rest. It must be quite a lot.

Catherine32 tries to look up the CPU stats but finds something mysterious… there is no CPU account! It doesn’t have an account because it’s not a real player, so all of its experience goes into the void. It plays so many games but doesn’t get any experience. Tough break.

Late that night Catherine32 gets a mischievous idea. She opens up her computer and creates an account. That account’s name? CPU. She does literally nothing for the next 7 days. The CPU account does the opposite of nothing, eating up experience through all hours via its robot slave. It becomes a behemoth, leveled above all others. Yet it is still under the control of Catherine32, and she is on top of the world.

Oops, it turns out players are looked up by username even if they’re an AI with no account. You can harvest the AI’s experience just by creating an account with the same username. This is a silly artefact left over from when there were no real player accounts and everything was just drop-in-drop-out.

All the players hide in fear, for nobody can hope to match the experience of the dreaded CPU. Until one day a new challenger arises, an account named Guest. It turns people play a lot of games when not signed in too. CPU and Guest become locked in an eternal war for experience supremacy, at least until I ban them both.

Unfortunately this didn’t actually occur because I caught the bug before release, but I wish it did. In fact, I was very tempted to just close my eyes and pretend I hadn’t seen anything. If you fancy hoarding some sweet experience yourself then sign up and play!

Instead, I could illustrate cards with simpler symbols and runes. This way I can focus on what I’m good at, which is the only thing that sets the project apart. This doesn’t mean I can’t have beautiful art in the game, it just means I won’t let its cost limit what I create. I’ll be able to make the gameplay the best it can be without worrying about budget.

The same holds true when considering scale. When you’re small you have few (or no) users, so you can’t play like the big guys. In fact you don’t want to play like them, they are the ones wishing they could play like you! If they could do what you have the opportunity to do on a small scale they most certainly would. Where they build anonymous matchmaking you can build community and genuine connection. Where they try to appeal to the broadest audience you can totally envelope a niche.

Your project has constraints. Don’t fight them, acknowledge them.