Macha

Any sufficiently advanced technology is indistinguishable from magic.

Arthur. C. Clarke

As a programmer, and a computer nerd, I obviously know more about how they work than if you just grabbed a random person off the street. As such, it's always interesting to see how people with less knowledge about computers use them.¹ To a lot of people, they click some buttons on the screen, and then the computer does something that may as well be magic, and finally results appear on screen.

However, it's not just those unfamiliar with technology that find that at some level, the processes going on may as well be magic. Very few people understand the entirety of how things work from top to bottom. Moving on from the unfamiliar user and the buttons they click, for many poorer developers they type (or copy and paste) in some code they don't really understand and somehow it all works. Their magic layer is lower than that of the user who doesn't understand how the program actually works. After all, they understand that programming code needs to be written, but they still don't understand any of what goes on beyond that. Their magic layer is off in the function calls, in the syntax on the language, in the meaning of the code. They may understand small sections, but the entire program as a whole is still magic to them.

Moving on from the poorer developers, novice developers, who instead of incompetence only have the problem of knowledge they don't yet have. They understand for the most part how their code works. They know how to structure their functions, they know when to use objects (and when not to), they can create programs from scratch. But they may not quite understand what is going on in the library routines, or what happens when the compiler creates a program from their code.

As you go further up in the skill of the developers, the magic level recedes. They realise that the library routines are just like functions they create themselves for the most part. Some might make system calls, but other than that, there isn't much magic there. They understand that the compiler reads their code, and produces machine code which is run by the CPU. The magic is banished from their code to the inner workings of the compiler/interpreter and the operating system.

Finally, once the developer learns how those last few retreats of magic work, they can understand pretty much the whole picture, as far as software goes. However, this still isn't the final end of the story. Sure, the magic might be gone from the software side, but there is still the hardware. How do those system calls translate to data being written to disk? How does the CPU know that MOV eax, ebx moves data between registers? And at this point you're in the hardware layer.

So, where is your magic layer? Personally I'm at the "inner workings of the compiler/OS" stage, and to work on pushing it further back, I've found some useful online resources. As a self taught programmer, most of the resources I've had up until now glossed over these areas, and it's my main aim for this year to push the magic layer back into the hardware. For compilers, I've found Jack Crenshaw's Let's Build a Compiler useful to start with, and I'm currently reading through that, and for operating systems, a bunch of useful articles have actual coincidentally popped up on Reddit and HN recently on that topic, though I'm still open for suggestons on other resources.

Some further discussion of this post can be found at Hacker News and on Reddit.