When starting as a software developer, everything seems like magic, compilers, IDEs, networking, kernels, etc. While we can ignore how these things work, to really grow as a developer, the magic must be removed. At the end of the day, our job is to produce something that works, having a well-rounded knowledge of how the full stack works can help you to understand when weird stuff happens, and come up with useful theories as to what is going on. Much as when I was a kid, I tended to take things apart to see how they work, and most times was able to get it back together; apologies to my Dad for the barometer/thermometer pieces he found one day. Here are a few things you might consider doing to “take things apart” and remove the magic.

Programming Languages And Virtual Machines

Make sure to know at least three very different programming languages. A combination of something low-level like C, where you deal with pointers and manage your own memory, one like Python or Java where a lot of things are taken care of, and then a language like Lisp. Look at more special purpose languages like sed, awk, perl, and m4. Definitely learn your shell – on Linux it’s usually Bash. Read it’s manual – you spend a good amount of time in the shell, and time spent learning it’s details will pay big dividends over time.

I highly recommend learning a lisp like Common Lisp or Scheme. Why? Eric Raymond says:

LISP is worth learning for a different reason — the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use LISP itself a lot.

I can personally can attest that this is true. Lisp has been around for more than fifty years, and for good reason. Lisp will be mentioned later below, so be forewarned. For a book to learn from, you could do a lot worse than Practical Common Lisp.

A great many languages run as either interpreter, or compiler to a virtual machine. Chances are fair that you use one of these languages, such as Java, or Python, C#, Ruby etc. Learn the format of the byte code for your virtual machine and hand code a hello world program in bytecode and get it to execute.

Examine the byte code output of the compiler for your language for code you’ve written and understand what you read.

Make it a career-long thing to learn languages that get buzz. Rare is the language I learned that didn’t make me better as a programmer, or provide unexpected utility, or expand my thinking in some way. You may get ideas on how you may be able to use some of the language’s ideas in the language you use in your day job. For example, you can write object oriented programs in C; whether or not this is a good idea is left as an exercise to the reader.

Learn about the lambda calculus. While turing machines are simpler for most modelling, a number of programming languages derive, some more, some less, from the lambda calculus. The article Programming With Nothing is a cool example of building FizzBuzz solely with function calls. The book An Introduction To Functional Programming Through Lambda Calulus is a very accessible book for this.

Spend a little time learning assembly. Write a few simple things and get a feel for how code gets executed on the metal.

Reading Source Code

You will interact with a lot of software if you do well in your career. Don’t be afraid to dig into the source code to see how things work or to patch things; especially things like your editor or IDE.

Networking

Read the RFCs for IP and TCP. While you may deal with other protocols (like UDP), the vast majority of protocols I’ve encountered use TCP/IP, it’s really good to know how it works, and what it does when it doesn’t.

Get a copy of Wireshark and use it to capture some sessions, and understand what you see.

Write code to use a raw socket that emits a ping packet, or an ethernet broadcast packet for an ARP request. You probably won’t be able to catch the replies in your code, but Wireshark will.

Kernel

Find out in broad strokes how the linux kernel works, and write a super-stupid device that resides in /dev and outputs hello world over and over. Consider writing a syscall and calling it from C.

Cryptography

Implement a symmetric key crypto algorithm in a low level language like C. Find ways to make it FAST. Compare your output against the known answer tests. Look at what the compiler outputs at various optimization levels and see where it’s smart and where it’s stupid. Figure out what you can precompute.

Learn about the pitfalls of implementing your own crypto. Learn why you really shouldn’t implement your own (except for fun anyway), unless you’re more than just a tinkerer; and you’re just a tinkerer unless you know better, and you probably don’t.

Make your own CA and set up a web server using TLS that requires connecting using client certificates.

Compilers/Interpreters

Learn how to write a lexical analyzer by hand (as opposed to using a generator). Learn how to write a parser by hand by using recursive decent. Learn about shift-reduce parsers and write a parser generator. The Dragon Book at least used to be “the book”. While pricey, it’s well worth it, and you could instead pick up the 1st edition instead.

Read Structure and Interpretation of Computer Programs, and write a simple lisp interpreter. If you hunt around, you can find the one page implementation in the Lisp 1.5 reference manual, which can make for a good starting point to get a feel for how the innards of eval works. Write a compiler for a virtual machine – either a vm you create or an existing one. Implement first-class continuations as you will learn a bunch of useful things in the process, even if you never use them in your day-job. Implement macros for your lisp and write something longer than 100 lines.

Event Loops

Event loops can show up in many places, whether embedded systems, hardware interface code, Node.js, GUIs and many others. These aren’t the only places where asynchronous code shows up, but they bring it’s utility into specific relief. The ability to basically have cooperative-multitasking in a single thread of execution is an enlightening experience.

Write a single-process, single threaded network server that uses an event loop to process requests for multiple connections, like a chat server or a (very) simple mail server, or something where the actions will deal with blocking operations to return a response.

Understanding and Wisdom

When learning all these things, strive for more than just being able to do it. Really strive to understand how they work, and why they work, and why they were designed the way they were.

Now that you’ve learned how all these things work, know when to use them, and how to write them, and now, more importantly, learn when not to.