Tensegrity

Getting to the first milestone allowed me to really start experimenting with long running Sky nodes for… well… really the first time. Well, at least for Sky. I had done a lot of performance work on Apollo, but that was quite a while ago. A whole lot has changed and new subsystems introduced. So while it was nice to have the local, in process simulation running, what I really needed was to get the docker images up and experiment with those. So with Milestone 1, I was able to get that started and – lol – immediately ran into some interesting problems.

So just puzzling through these issues was quite interesting, of course. There were a small number of complete WTF moments where – for example – I was using the number of units in an epoch for the number of epochs, which – when discovered – immediately snapped explanations for head scratchers that I had been seeing for a while. But there were some interesting issues that were fixed at the foundational levels so not exactly a trivial bit of work on the Apollo side, although it was largely unnoticed at the level of the Sky Application.

Resevoir Sampling

One of the things I fixed was the ResevoirSampler I use in Apollo. This was, needless to say, a rather nasty and subtle issue as this class is used to sample the large populations that Apollo is (theoretically) supposed to deal with. If you’re not familiar with the theory of resevoir sampling, I highly recommend it as getting statitically random samples of large cardinalities and streaming data is a very interesting problem. In my case, I had not really implemented the correct algorithm and… well. A rare edge case would create the rare flapper and… yea.

The problem that I was seeing was one of replication. The form of state replication used in Apollo relies on exchanging Bloom Filters indicating state vectors and replicating based on the differences between these vectors of two nodes in the cluster. One of the tricky things about doing this replication is that one usually wants to limit state transfer to reasonable sizes. In Apollo, it would certainly be possible for members joining a large cluster to be swamped with the amount of state that needs to be transferred – such as the membership list, or such.

In Apollo, the amount of state transferred is limited per call to a reasonable number and so the issue becomes how does one sample the (potentially) large amount of state to see if it needs to be transferred to the gossiping partner? So I hopefully got that figured out and this state sampling pattern will scale to 3-6 orders of magnitude of membership sets. We will see, of course. Once that subsystem clicked into place correctly things stabilized quite a bit.

Ab Initio

So there was a lot of fixing going on. I did some tweaking of the Fireflies Join protocol and made that process parallel and asynchronous. That was a huge success and sped up testing, too. It was, of course, also subtle and quick to anger, but I had fun pulling through that one late night. I had to debug a couple of tiny nightmares in the Fireflies View change finalization process. Likely a germ or two still lurking in this, of course. Which is why I want to do the long running testing, of course, but… lol.

In the end, I had to refactor the BFT subset logic. I’d like to share that between the DynamicContextImpl and the StaticContext, but I’d have to abstract the ring out, I think and I’m too lazy to do that at the moment.

There was also some interesting concurrency issues debugged along the way. One of the bedrock, early design decisions of the critical Finite State Machine subsystem, Tron, is that the actual state transitions occur atomically. It’s not the only pattern one can imagine, but my tiny brain cannot really figure out how to make that with minimal lock duration for example. The upshot is that the execution on the state machine of a transition is synchronous, blocking and requires exclusive access to the state machine.

I think this is a natural model, but it does have a number of subtle issues when these state machines operate with multiple asynchronous transitions occurring in parallel on these state machines. If you don’t watch it you can get yourself into a deadlock because a needed transition cannot occur because the state machine is already locked by another transition thread. lol. lmao. Anyways, long time problem so I cleaned up a lot of the transition implementation actions and whooooo boy did that clean things up significantly. Just this alone – lol. So that was satisfying.

One of the more subtle issues I found was my use of the number 7 in configurations for the epoch length in Ethereal. Another SMH moment, to be sure. The explanation is rather beyond me to explain simply at the moment, but the problem has to do with the way the Aleph asynchronous BFT consensus works with a round delay of 3. Anyways, there’s some interesting edge cases around the chain RBC protocol termination of the epoch that causes a stall with the number of units <= 9 for an epoch. So, fixed that and require 11.

Long(er) Running Testing

Once I had cleaned up a lot of the aforementioned issues n’ such, I was still running into this weird ass memory leak and then sudden stall and then eventual process death. That problem turned out to be the way H2 database implements binary data. I, of course, was modeling the necessary database stored for the various binary things Apollo needs to store with VARBINARY. Apparently the implementation in H2 is beyond my understanding and I managed to get a conflicting lock for what appears to be a long running transaction.

Or, at least the BLOB thought so. Investigating through google showed highly similar issues with others and some references to caches that H2 uses to make this stuff performant. So, whatever I needed to be doing to use that correctly with multiple threads, well, I wasn’t doing. Nor do I even know what I was doing wrong at this point. Because reasons, I guess. Anyway, I first thought the issues was not using fixed length, but that turned out to not be the case and so I ended up converting it all to strings and base 64 encoding it because I don’t care at this point and it certainly solved my problem.

What was hilarious was how I found the problem in the first place. As mentioned, I ran into this stall when processing. Sooner or later it’d hit no matter what I’d do. The more I fixed, the first and more random it got. What I found was causing the problem was the cache expiration I had programmed into the KerlDHT through it’s use of the CachingKERL. Basically 10 minutes. Like a shotgun that would hit across the cluster – or close to each other, of course – and when these entries were flushed, Ethereal would come to a halt because – well – it needs these keys to verify signatures, etc. And with this H2 behavior around BLOBs the lookup to refresh the entry was…. literally hanging and timing out after forevs. Just kind of ugly. Anyways, good riddance! Strings work fine for my use.

Lottsa Blocks

So you can now spin up Sky nodes for the local demonstration as described. There are issues with a lot of new nodes joining, but if one limits to 2-3 at a time, you can build up quite the cluster. Right now it doesn’t do much, of course, as there are no external transactions input into the system. And due to the way CHOAM works, if there are no transactions in the Ethereal Preblocks, then there’s no need to emit a CHOAM block. So if you spin that demo up you will see blocks periodically produced, but not super fast ;).

The blocks you see after the genesis bootstrapping or synchronization with the chain are due to the periodic View changes that happen in CHOAM. Approximately 30 preblocks are elided for every RECONFIGURE block you see produced. These 30 elided preblocks would, if they had contained transactions, produced CHOAM execution blocks. You can see these produced if you tweak the logging settings, of course. But it works out to quite a few blocks even with my relatively conservative settings I have configured for the testing and docker local demo.

I ran these clusters for about 12 hours now and they seem stable as a rock. I ran this under local simulation as well, under the java profiler. CPU load, after all my fixin’s, is extremely low. Memory seems nicely bounded and very low per node in CPU as well. Have run over 30,000 blocks now in a 20 member cluster and so far haven’t run into any issues at all, so fingers crossed.

So now that things are much more stable I can continue the work on the Oracle API I was adding to Sky. Once that is up, I can target Milestone 2, which is replicating the Oracle smoke testing that’s done in the local simulation testing with the Docker local demo. That’ll be truly hilarious to witness working. Been waiting a long while for such a thing – lol.

Anyways, things are surprisingly stable and functionally complete at this stage. Ima enjoying it while I can.