This week I made huge progress... with very little on screen to show for it :)

Like last week, this week was spent on more tool/engine stuff and, in particular, what I now refer to as the "execution engine" part of things.

The execution engine is basically the all-encompassing term for the operator graph/compiler/JIT. It's basically responsible for everything that executes dynamically on the CPU. And, as luck would have it, it's already working, super clean, super fast, is already drawing content on the screen, and has a basic editor!! :) Of course, all it's actually drawing is a cube (which is the post screenshot), but it shows a proper vertical running through all the new tooling stuff, and that's pretty rad. Basically the same thing as the infamous black triangle, but, well, looks a little different.

After fighting with some JIT details last week, I decided the most fun path forward was to continue on that route and go for the whole JIT thing early! And boy, am I glad I did. After lots of thinking about the whole system I realized I could basically implement all of the rendering engine in Rust still, but expose a C API of sorts to it, and then just compile the operator graph into some glue code that moves some data and calls into the engine. Sounded simple enough, so I started by doing some isolated testing where I would dump some asm into a buffer and execute it, but unlike last week, this time I was trying to get it to call back into Rust code. For extremely basic cases this worked fine, but as soon as the functions I called into were calling other functions, things got wonky.

I did a bunch of testing in Hopper and found that when I was calling functions like printf (mostly indirectly through Rust's println! macro, but I also tested calling the raw C fn just to be sure) eventually, somewhere down the stack, things would blow up on some SSE instructions. And it wasn't all the time, either - it was consistent to the extent that a piece of code would always work or always fail, but if I changed things like call order of the code or number of arguments to various functions, the results were somewhat unpredictable.

After lots more testing and disassembling similar functions to what I was doing written in Rust, I narrowed it down to the prologue/epilogue and the stack being misaligned somehow, but I was having a hard time seeing how exactly. Luckily, I popped into the Rust users forum, and they were helpful as always - on 32-bit OS X, esp has to be 16-byte aligned each time a call instruction is executed. Without going into too much detail, this is basically to simplify how memory on the stack will be allocated for SSE variables, which have to be 16-byte aligned. Having a convention like this guarantees that compilers targeting this platform can simply use constant stack offsets for SSE var's, and as long as all the other functions follow these same rules (... :D) it all works out. So this was a pretty easy thing to fix, and after some more stress testing, seems to have solved all the issues I ran into.

Interestingly enough, even though I had done some x86 assembler by hand on win32 before (for basically all my 4k's from 5+ years ago), I never ran into this problem because win32 expects 4-byte alignment of the stack pointer instead of 16. Funny thing is, this mostly "just works" since most function parameters will be 4 bytes in length, and I guess that covered just about all of the functions I ever called into back then... :)

The best part of all of this, though, was that at this point, I had basically done enough research to figure out all of the rules I had to follow to make this work and more or less proved that it's possible to do a JIT like this that will work fine on both OS X and Windows, which is important, as I don't want to have separate backends for the tool and the intros.

Getting it to actually draw stuff was extremely easy at this point - it was basically a matter of replacing the interpreter code I had from before with either emitted assembly or calls into functions that were basically identical, so it was mostly just moving code around. But man, the feeling when it actually worked was absolutely euphoric - especially when it worked first try in the tool on both platforms! To quote me, 2 days ago:

ferris [10:35 PM]  
I just yelled out loud

[10:35]  
punching the air all up in this bitch

[10:36]  
hahahahaha

[10:36]  
YESSSSSS

[10:36]  
we are JITTING IN THE FUCKING TOOL BOYS

github BOT [10:38 PM]  
[demotivation:no-nodes] 1 new commit by ferris:
`f2ebd5b`  HOLY FUCK WE'RE JITTING IN THE TOOL - ferris 

ferris [10:38 PM]  
my jit is drawing a cube

[10:38]  
I'm freaking out

[10:38]  
aaaaaaah hahahaha

[10:39]  
bout to test on windows

[10:39]  
if it just works.. oh man (edited)

ferris [10:45 PM]  
omg

[10:45]  
it worked

[10:46]  
FIRST FUCKING TRY ON BOTH FUCKING PLATFORMS

[10:46]  
guys this is a HUGE win

[10:46]  
:smile: :smile: :smile:

[10:46]  
I feel like crying hahaha

[10:46]  
seriously SMASHING this, not even 2 weeks after the last release

[10:46]  
https://github.com/yupferris/demotivation/issues/87

[10:46]  
GUYS (edited)

[10:46]  
!!!

[10:46]  
:smile:

ferris [10:47 PM]  
[same image as the post image]

ferris [10:47 PM]  
that blue box

[10:47]  
is a cube

[10:47]  
rendered by Rust code

[10:47]  
that's called by x86 code

[10:47]  
that was generated by two layers of compilation

[10:47]  
from a model that represents two boxes in a grid

[10:47]  
onto a framebuffer that comes from qt

[10:48]  
guys this will fucking work

[10:48]  
AAAHHHH

Sooo yeah, I was a little happy :)

And seeing as I had gotten this far much earlier than I had planned, I took the opportunity to work on the operator editor UI a bit, and came up with this:

It looks like shit and all you can do with it is change which op is the "master" op (where execution starts), but it works with full undo/redo, triggers lightning fast recompiles, and generally feels pretty good (and I got to remove/clean up even more code from the last tool iteration while putting it all together!). Next steps will be to implement things like selection and moving/creating/deleting ops, serialization/deserialization, and of course the inspector, where the operators' expressions will be presented/edited. So, let's see how far I can get this next week :)

Until next time!

P.S. After writing this, I counted up all of my everyweeks entries, and considering I now have 51, and I've only missed one week during that time (iirc), that means we've been at this for 52 weeks now. Which also means our everyweeks endeavor is now A YEAR OLD. So HAPPY BIRTHDAY EVERYWEEKS!!!!! Here's to another great year of growth and radsomesauce :D

Last Edited on Sun Aug 07 2016 14:25:33 GMT-0400 (EDT)