Luna-Canis

So after taking almost 1 year off, (big diversion in looking at finance stuff), I started looking at the scripting problem again. What’s good about having stopped is that the proper solution is clear. Start with the scripting language first. Start with that before anything, from that we get everything, expressions, shading language, proper objects, aggregated data types, namespaces, everything. I need to start a new scripting language seperately, implement a new type system, and start from scratch.
Also, I need to focus on the pass by reference/value/pointer issue. Wasn’t solved in the original implementation, so now I have a chance to squash it.

Here is a spec:

Grammar:

• Recursive

• Namespaces
• objects/classes/constructors

Internals:

• Reuse the letter/envelope idiom used for spa. (Was very effective)

• Has axiom data types. Int/Double/Enum/Function/String/void/etc.
• Some contructors are purely script based, but some constructor will have a c++ component as well
• Type dictionary
• Gets compiled into bytecode.
• compiled bytecode is executed, and manipulates a stack
• Objects define themselves as being copied by value or reference.
• handle references, maybe pointers.
• typesafe
• Has an execution stack, and heap.
• Any non-axoim (binary) objects are callable, with a return type.

Got the N-Tick working. To get this operational we had to have the FinNode and the TickData a way to pass event data along side the finance data. We got that working, so he current state can display the n-tick time and value. The Event class is flexible enough to mix in different events of different types.

The next step is the Instaquote IPLink project. I got source files to build and run a windows app for connecting to Instaquote and getting financial data directly from them. I need a username/password from the Instaquote guys, so I’m waiting on that.

So next is to get the Expr node. The last thing I had to get working is to find a way to create spare parameters. The current method only allows for arrays of doubles. We want a gui to be able to specify any kind of spare parameter. To do this, we need to extend the types of passive gui elements under Composite.hpp. We need a Label only and a Text only.

Had to stop to do some overtime work. BUT, the overtime labor paid off, and I have a new laptop to do work remotely. VERY psyched about it. Overtime is over, holiday season is starting, but I’ll be able to sneak in work a LOT more often.

Where I last left off, I decided to stop doing the generalized expression node, and just to implement the n-tick algo that Chris suggested. Got that working VERY quickly, good. But it also allows me to focus on another hole in the API. I need to be able to define and pass on events to be eventually displayed.

Also, I realize that we need some support for navigating bwain using a laptop, multitouch interface. Modern window libraries most likely have support for multi-touch mouse pads, but the version of Qt I’m using is probably about 10 years old….

Also, NEED to find some free(cheap) online hosting for the source. Its the only sane way to share code between both computers….

Very psyched about this. This is a great example for the expression language to make a profound impact. To make it work, need to implement some missing pieces:

• Need a multi-line string input to define the expression. Actually, all I need is an array of strings, then have a new active gui widget to handle it.
• Need some variables defined. All these variables have relative addressing
• delta[int] - 1st order difference between current sample and last sample
• sample[int] - tick data. This should be readable and writeable
• time[int] - timestamp. This should be readable and writeable.
• The timestamps are writeable, but shouldn’t be allowed to change the tick ordering. But how can you enforce that? Maybe provide a sorting operator or something? Still, no sense in constraining it, or detecting it.
• Allow for any local variable to be automatically binded as non-varying inputs.
• Have to improve the temp variable dialogue. Need to be able to define vector, vector, string, toggle (int), and menu (int).

If I can get this working, this will bullet-proof the shading language.

I forgot all the stuff involved with introducing a new Qt widget through the system. The multi-line edit is QTextEdit, have to subclass in guq first to bridge the signal-slot bullshit. Already got a text instanced. Since we’ll be coding, I have to allow for ‘return’ key to be pressed and keep the focus. ‘Shift-Return’ will remove the keyboard focus.

Got the QTextEdit working with the guq library. Next is making it an active parameter widget with the guqLink library.

Wow, got the widget instanced as a parameter. Have to work on getting it to play well with the application. Its not catching the up/down arrow events. Also, the Shift-Return is actually inserting a ‘return’. Then I have to get it to update the Array properly. But a lot of the hard part is done.

Ok, now I need a strategy for saving the multi line widget contents to an array of strings.

• Once the user is done editing a line (Qt calls it a paragraph), we write the line to the string array
• Ways to detect when a user is done editing a line are:
• Hitting ‘return’. Return also inserts a new line, potentially splitting an existing line.
• using up/down/left/right arrow to go to another line.
• So optimally, we need a callback signal to tell when the user has finished editing a line. If the guq::TextEdit could handle this functionality, then that’s less work for a client to figure out.

Got this all working… Next up… more types of tmp variables. Should start with something minimal for the moment. But eventually, we should be allowed to specify any type of tmp variable type, and any type of gui to control it….. For now, just arrays of floats, ints, and strings will be enough.

Got the basic display mechanism working. Its very similar to the image processing model. The data is read in pieces, collected into a master buffer and displayed. I can properly display one array, and I’m trying to get the FinViewer to display multiple stacked arrays, each with their own view. That might be the most compact/accurate way to display multiple data sets since they have such varying ranges. I thought I could display them in one view, but that doesn’t work at all.

So the task is to detect for a condition, and flag it. Thinking about how I’m going to represent these ‘events’. Maybe just going to call it an ‘event’, and the data streams pass arrays of samples and events for display.

I’m happy that I’m going to get the shading language involved to make this happen.

Enough with the display. I can display multiple traces, display their values, edit their individual views, etc. I still can’t think of a solution for doing the tick marks in time. I need to find a ‘niceNum’ that works for years/months/days/hours/mins/secs.

Starting on the expression flag node.

Had to take time off to deal with some more immediate financial issues….

If I recall, I was starting to define how the data flow was going to behave w.r.t. the ScalarArray datatype. The whole setDirtyBit/cook order. I’m using the imgNode system as an example.

• The source node has to be able to define its ScalarArrayBounds with a minimum of cooking.
• This is passed down through the operators to the Buffer node, where it keeps a master list of the entire ScalarArray for display.
• Any time a node has a parameter change, it calls Node::setDirty(), which triggers updates to all its output nodes downstream by calling setDirty() on them.
• ImgNode::getBBox() is used to get the image bounds. FinNode::getBounds() is the equivalent.
• ImgNode::fillTile() is used to calculate a piece of the image. FinNode::calcData() is the equivalent.

Got the Buffer class, and all the execution methods filled in.

Start on the Fin Viewer! Need to be able to see these samples. I know I shouldn’t go too crazy in implementing this. Just need the basics to be able to see these samples, scan across them, etc. I keep looking for examples of other financial software online…. Just tick marks I guess….

At the very least, I would need a way to make tickmarks. The values can be done with niceNum() (Graphics Gems I), but for time, I would need a niceNum() for financial time. Something that will handle years/months/days/hours/minutes/seconds.

Got the file to read in the data. Working on the display next.

There are 3 instances of a 2D gl view, one for the channel editor, one for the image viewer, and now a 3rd one for the fin viewer. I REALLY should create a GL 2D palette class. This would take/save views from a BBox, then subclass this for the the image viewer, and the fin viewer. I guess I have the time. I really don’t feel like re-writing all the get/save 2D view stuff….

Ok, maybe I don’t have to change the other two existing widgets. That would take a while. But for the FinView, I’ll start with the 2D canvas widget first, then have the FinView subclass off of that. I can look into using it for the other classes later. And I’ll have it available for future classes.

I can’t (rather shouldn’t) call it CHOPS. Since what we’re doing is processing arrays of scalars, I’m calling the datatype, ScalarArray, and ScalarArrayIn. ScalarArray is for homogenous data, ScalarArrayIn is for inHomogenous data, and includes an extra array of timestamp data. This is what’s going to represent the tick data, but calling it ‘TickData’, would have been too restrictive.

I’m starting the Buffer class. I guess I should call it a ScalarArrayBuffer. Like the Image Buffer, it drives the interactivity and the data flow of the entire process. I’m starting with a simple file reader. But when we start looking at real-time feeds, this node will be instrumental in behaving as a queue, pulling in more data as it becomes available.

New Tech! Is always interesting.

Going to begin to write operators to read/process financial data. This link, points to some good raw data to start with.

Biggest issue so far, timestamps. I need a timestamp utility with usec granularity. The Boost stuff is all good, but it tries to be a bit too clever. I need time data that I can interpolate, use for calculations. The Boost stuff is too symbolic, as opposed to numerical.

gmtime seems to do the trick. I’m going to have to do some testing to see if I can initialize the structure and get the correct results. I’ll need that, plus another 16-bit short to count the milliseconds. The tick timestamps seem to have millisecond granularity. I think there are also methods for converting to different time zones, which could be very useful once we start dealing with exchanges in different time zones.

Looks like I already started a Time data type. It has the proper granulatiry, plus some numeric operators. If I can initialize it from an ascii date/time, I’m in business. ‘mktime’ is what I”m looking for….

I was able to get everything working by using the boost library stuff. I calculate the unix time myself, once I found the proper examples, the entire library fell into place. Timestamps are all good.

So I’ve added two new data types, ScalarArray, and ScalarArrayIn, for homogenous, and inHomogenous scalar arrays of data. The first two nodes will be a read operator, and a buffer type operator. Then a display node.

Got the timestamps fully working. Found a great reference here, that was very helpful in calibrating the timestamps, converting them from unix time, to boost::date_time, and back. Its solid now…

I’m getting stuck in how to implement assignment operations. Specifically, things like m[0][1] = abc, things like that. The way I have it now, I would be copying multiple instances of m onto the stack, then having the [] operators write their part, then copy all of ‘m’ back, which is a complete waste. What if ‘m’ is some huge 2D array? Beginning with ‘m’, that should return a reference, not a value.

How do I encode this using the existing type system?

spa::Value can have a boolean ‘ref’. If its true, then it doesn’t own its own data. spa::Value has the option to do a deep copy, which would mean owning its own data if necessary.

Got the 2nd proof of concept working! (The first proof of concept being the failed design I gave up on…) The simple example I started on works! Advantages to doing a proper code flattening: faster, easier to debug, can save out a pre-compiled script, also easier to do things like throw-catch, easier to look at the run process.

Now I’m left with having to implement the script commands for all existing funct types.

Crud… this is going to take a while… and even longer to completely debug….

Also, going to have to find some trickery to get references of subTypes. Previously, I was copying it, but with references, its trickier. ValueCon<> is a subclass of ValueImp, I could try creating ValueConPt<>, which would point to its data, but not own it. That will work… Also, that means that I have an implicit pointer implementation. Whoa… With pointers, I can randomly access packed arrays of data, which is kind of what the bracket operator does. So its actually more of a reference than a pointer. BUT, what if I allowed pointers to alloc/dealloc their own memory? That could be really cool, or get very confusing. A scripting language with its own heap space. Would that be a first? Also, I don’t know how the type system would handle it. It would be cool if I could keep chaining pointer modifiers, like int****, but not possible, and completely unecessary. So at the most, any value could be a reference/pointer.

Ok, this is definitely a diversion, but I would need to really test this stuff before I break everything.

Change of plan, not going to do it. Just going to keep the existing reference method in place. All bracket operators will copy the contents of the slice on the stack. If its too slow, I’ll deal with it later. Maybe have to create an iterator class to deal with this. The method I mentioned above is too much of a hack.

This is the most fragmented blog entry. I decided not to erase my train of thought. So here is the latest attempt. Using the Type mechanism, I’m going to try to make a reference type. So we can do Ref or Ref or anything. Hopefully, I can find an easy way to make it work…

Another interesting thing. The bracket operator has different meaning depending on when its part of a lhs expression or a rhs expression. In a rhs expression, its read-only, so the value can be copied. For the lhs expression, that’s when we need the reference functionality. So we need to be able to flag whether we’re in a lhs or rhs…..

There is no way around it. I have to flatten the code, the smd::Cmd and spa::Funct have to execute in a single stream of instructions. Two stacks, one for manipulating symbols, another for local variables. In keeping with the simd model, the Funct class will have some static method that can be added to a list of script functions that get called in succession.

• Instructions can have state, but they should be kept with a list of spa::Value in the superclass. This way, the instructions will be easy to copy. That also means that we wont be dealing with virtual function calls, which will keep the execution time optimal.
• Symbols can have a name, and probably should for debugging purposes. But ultimately, symbols should be referenced by index.
• The really good thing about this is that it will be trivial to copy compiled scripts around, and they can optionally be saved in bytecode form. I don’t need this feature yet, but it could be handy.
• It would be nice to be able to just re-run the simd virtual machine as a single thread to run the scripting language, but we would lose the symbol abstraction. Also, the simd virtual machine is register based and can’t handle recursion. The script virtual machine is stack based.
• One question, should the instruction stack be decoupled from the symbol stack? An instruction push always implies a local symbol table, but a local symbol stack does not imply an instruction stack. I would have more flexibility if they were decoupled.

Ok, a lot of work to do. In the end, it will be a WAY cleaner implementation. The variable binding business with VarValue was getting messy. Looking forward to having the recursive example work properly.

Of course, the only thing harder than designing an API is coming up with an example that’s simple enough to run as a first test….

Ok, here’s something funny… if I had started writing the scripting language first, I could have had the virtual machine create simd code as well. I would have been able to share a lot more code base that way. As it is now, I’m duplicating a lot of functionality. Live and learn I guess. Writing the simd virtual machine was a lot harder though. Maybe its easier doing the hard part first, then connecting later.