Luna-Canis

So I tried the following:

void ctor(int i) { print “ctor ” i “\n”; ctor(i+1); }

ctor(0);

I ran it, and it only recurses 757 times before it kind of stops. Its odd, I can’t tell where its failing, it seems odd to me that it would just run out of RAM. I tried running a similar function in perl, and it just keeps going. I tracked the RAM use, that’s definitely not the issue…

I narrowed the point of failure to a single deque assignment. I’m not too sure as to what would cause an stl function to silently end the process. Maybe memory is thrashed. In any case, I suspect I’m doing too much processing for each function recursion. I need to list exactly what happens when I’m doing a function call and see if I can’t streamline it.

I found this, explaining how stl will throw exceptions when they have memory failures, and possibly ways to alleviate it. When I run it, the task manager only report the process at 20M, which is nothing. Don’t know why stl would be failing. Also, the runtime stack is WAY deep. The debugger wont (can’t) display all of it. Each function call adds about 4 layers to the runtime stack. Given that the script is at 760 calls deep, that’s about 3040 runtime stack calls. But perl still works really well. Maybe I should try a release compile and see how far that gets.

Did a release build, and it executed the function call 3890 deep. Still, not as efficient as perl, but I guess it does appear to be a ram issue. Tomorrow, I’ll work on optimizing/streamlining the function calls. Crud… I ran the perl example until it ran out of RAM, it went 2262191 deep…. Python defaults to a stack depth of 1000, but when you extend it, it runs even deeper than perl.

Looked around online. To do a scalable, production ready scripting language, requires flattening the code into opcodes, and running a virtual machine in a similar way that I’m running the simd virtual machine. I’m not going to get into that now, I just want to get it working. Speed and scalability will have to wait. Found this book that explains a generalized virtual machine implementation.

I’ve been using the debugger to trace down the cause of the bad_alloc. I’ve narrowed it down to a SymbolTable constructor. But that doesn’t solve anything. The SymbolTables have been working forever, now that I’m allocating thousands of them, they’re crashing. I’ve noticed that there is a LOT of overhead for the symbolTable base class. So the next step definitely should be a streamlining step. There are many std::map and std::multimap’s being created unnecessarily. The symbol table for the stack should be a lot lighter, if its going to be instanced like that.

Each invocation of the execution stack doesn’t need an entire symbol table. All it needs is a list (deque) of spa::Value to hold local variables, and that’s it. The variable names have already been resolved during execution. Short of creating another virtual machine, this will save a LOT of computation and RAM overhead.

I got the new execution stack setup implemented, and it STILL fails at roughly 750 calls deep. WTF?? And its still using roughly the same amount of RAM. I should try compiling a binary to just run the scripting language, with no application or gui. Just to see how far that gets. But I should probably let this go, can’t figure out why its throwing a bad_alloc.

Ok, I’m not letting this go. I can’t risk having future memory corruption happen because the scripting language is unreliable. I’ve come up with some strategies to isolate the problem. I tried just stacking an array of spa::Value objects. That just keeps growing with no problem, so spa::Value isn’t it. I also simplified the script function itself. I’ve come across a sloppy problem with VarValue. I should allow for VarValue to be constant, and not be required to be binded. That way, the function symbol doesn’t have to be updated. Also, what could help is to add a function call to the grammar. Also, VarValue has a ’sym’ pointer that should be cleared before executing as a script. Just some cleanup that I’ve come across while simplifying the problem.

I’ve tried to grow the virtual machine execution stack arbitrarily deep, and it seems to be working. That doesn’t leave much else except for the expression evaluation. To test, I’m going to add a function call to the grammar, instead of having it being invoked as an expression. If that works, then I know to focus on the expression evaluation.

I FOUND THE FKING PROBLEM…

The execution stack size defaults to 1MB. I can change the stack size in the compiler, but maybe that’s not the way to go. I thought that the stack would just keep growing into the heap space until the two met, but I apparantly not.

SO, that means that I have to do more work on the virtual machine. Unfortunately, that also means that the way I’m evaluating expression wont work for the scripting language. I hope I can avoid doing this, but its looking like I may have to edit each Funct class again to operate on a stack. That at the very least, I may also have to resort to flattening the code as well. The SIMD engine does all its operations in a single loop. Optimally, the scripting engine should be able to do the same thing. But that’s a LOT more work… ugh….

I’ve been giving it thought… don’t think there is any way around it. I’ll have to add a method to all Funct operators that will execute themselves on a stack instead of using the c++ execution stack. Now whether that will require for me to flatten all the code is another question, I’ll see what I can get away with….

There is no way around it. I have to flatten the code, the smd::Cmd and spa::Funct have to execute in a single stream of instructions. Two stacks, one for manipulating symbols, another for local variables. In keeping with the simd model, the Funct class will have some static method that can be added to a list of script functions that get called in succession.

• Instructions can have state, but they should be kept with a list of spa::Value in the superclass. This way, the instructions will be easy to copy. That also means that we wont be dealing with virtual function calls, which will keep the execution time optimal.
• Symbols can have a name, and probably should for debugging purposes. But ultimately, symbols should be referenced by index.
• The really good thing about this is that it will be trivial to copy compiled scripts around, and they can optionally be saved in bytecode form. I don’t need this feature yet, but it could be handy.
• It would be nice to be able to just re-run the simd virtual machine as a single thread to run the scripting language, but we would lose the symbol abstraction. Also, the simd virtual machine is register based and can’t handle recursion. The script virtual machine is stack based.
• One question, should the instruction stack be decoupled from the symbol stack? An instruction push always implies a local symbol table, but a local symbol stack does not imply an instruction stack. I would have more flexibility if they were decoupled.

Ok, a lot of work to do. In the end, it will be a WAY cleaner implementation. The variable binding business with VarValue was getting messy. Looking forward to having the recursive example work properly.

Of course, the only thing harder than designing an API is coming up with an example that’s simple enough to run as a first test….

Ok, here’s something funny… if I had started writing the scripting language first, I could have had the virtual machine create simd code as well. I would have been able to share a lot more code base that way. As it is now, I’m duplicating a lot of functionality. Live and learn I guess. Writing the simd virtual machine was a lot harder though. Maybe its easier doing the hard part first, then connecting later.