WaspDoc Bug Fixes

Looks like there were a couple TConc-related bugs lying around in the Waspdoc argument parser. Fixed for posterity.. Perhaps I will eventually get around to finishing waspdoc/check-source.ms and waspdoc/c-file.ms.

Fixing Static vs. Dynamic

Courtesy of Chris Double, Wasp's build system will now statically link libevent on Linux and other Unixen for the stub virtual machine. This improves the portability of standalone executables compiled using Wasp's stub virtual machine, since libevent is not quite ubiquitous.

Also patched was an annoying little warning in "vm/salsa.c" related to an implicit pointer cast, and the process loop now performs a 100 microsecond sleep when all processes are polling. (Previously, this was handled by the OS process, which doesn't correctly handle the possibility of more than one process polling for external events.)

Portable Terminal I/O with SCUD

A new release will be uploaded, today, 0.4.1, with some minor bugfixes and a simple terminal abstraction library, "lib/scud." This library supports cursor relocation, changing terminal colors, and clearing the screen on both VT100-capable terminals and Windows. Also, a new primitive has been added, "unbuffer-console", which puts the Windows and UNIX standard input into a raw, unbuffered mode. This makes it possible to capture individual keypresses by the user without waiting for linebreaks.

Why the fuss with terminals? Because I have been working on a lot of console mode applications, and this module gives me the bare minimum while supporting both Windows and UNIX platforms. There is still a lot of planning and work ongoing with 0.5, but not much of it is suitable for release, yet.

Happy holidays!

Kicking Wasp 0.4 Out of the Door..

This has been languishing in my To Do list for the past couple weeks, introducing Wasp Lisp 0.4. This version provides a lot of little improvements in the network and console I/O on Windows and Linux, and, more importantly, contains a complete port of MOSREF 2.0 from MOSREF 2.0b3. All of the functionality we demonstated two years ago from MOSREF is in this newest port of MOSREF, but with different ciphers. Stability and speed have greatly improved, compared to the 2006 prototype, largely due to the use of the Salsa/20 stream cipher for the session cipher.

Work will begin soon on Wasp 0.5, which consist of more primitive namespace cleanup, bug fixes and, hopefully, WaspDoc.

Link Dump: Plugins and DLLs

MOSREF requires the creation of statically linked "monolithic" binaries for Drones. Wasp, prior to 0.3, supported loading plugins written in C by bundling the entire virtual machine into a shared object file. The following is a list of links I have been looking at, trying to find a way to support plugins while keeping the virtual machine itself in the executable:

• FlexDLL -- Emulates UNIX dlopen on Win32.
• EDLL -- Reimplements the dynamic linker for Windows; also catalogues common workarounds for the DLL plugin paradox.
• DLL Creation in MinGW -- A succinct tutorial on LoadLibrary / GetProcAddress.
• The University of Colorado MinGW FAQ -- Contains a nice list of common MinGW gotchas.
• The Program Library HOWTO -- More about how to design libraries on UNIX systems than Win32, but discusses those techniques thoroughly.
• Wikipedia: Dynamic Loading -- Contains an interesting technique for DLLs to access the original process's symbols.
• Dynamic Linking in Linux and Windows, part two -- More in-depth discussion of DLLs and their limitations.

The criteria for a successful solution:

• Permits a plugin subsystem to call routines in the virtual machine. (Blocks naked use of DLLs and LoadLibrary / GetProcAddress.)
• Shares data structures in the virtual machine process with all plugins.
• Does not bloat the virtual machine.
• Does not require special effort by plugin authors. (Disqualifying passing a structure of API function pointers to the plugin.)
• Does not tightly couple the plugin to the binary. (Disqualifying various --output-implib tricks.)
• Does not excessively complicate the build process. (Disqualifying LibTool.)

MOSREF Shell Shock..

Work has been quiet but steady on MOSREF, with some very intense work on restoring the "proxy" command and improving the Console's resilience against malfunctioning Drones and noise. All of the functionality from MOSREF 2.0b3 has been restored, with the sole exception of the "scan" command; it took me longer than expected, but a very subtle bug in vm/os.c was locking up Drones when a specific sequence of proxy sessions was introduced. I also discovered an interesting hiccup in GCC when compiling vm/format.c with -Os or -O3. Until I get a chance to examine the optimizations involved, the Makefile will default to compiling with the more conservative -O2 optimization flags.

Progress on MOSREF..

As I have mentioned in the last few posts, MOSREF is being ported to the Wasp Virtual Machine. There have been a few changes in this new version, mostly the reduction of code complexity, and some improved handling of man in the middle attacks. Previously, each message had a hash used to detect transport compromise, but this did not prevent a permuted message header from tricking the Console or a Drone into waiting an indefinite amount of time for a ludicrous amount of data. The new model uses a two-tier checksum model; each message is composed of zero or more blocks. The message has its checksum, and each block has its own checksum. To increase the difficulty of spoofing valid blocks, the block sizes are randomly sized, which ensures that the offset of the third and successive CRCs are difficult to predict. Next up, getting the SOCKS 5 proxy working again and further code complexity reductions.

The Best State Machine is a Coroutine..

Occasionally, people will ask me why I think Wasp is better than other Lisps for network programming. One of the big reasons is its pervasive use of lightweight processes as coroutines, a technique often used by Erlang programmers and, occasionally, advanced Scheme libraries. An example, from "lib/collate-filter":

;;; -- SNIP --
     ;;; "in" is an input channel, waiting on it causes a process to block.
     ;;; "out" is an output channel, which connects to the next coroutine.

      (define buf (make-string))
    
      (define (wait-for-any)
        (forever
          (define evt (wait in))
          (when (string? evt)
            (string-append! buf evt)
            (return))
          (send evt out)))
      
      (define (wait-for-field len)
        (while (< (string-length buf) len)
          (wait-for-any))
        (string-read! buf len))
    
      (define (wait-for-int)
        (string->quad (wait-for-field 4)))

Wait-for-int causes the process to idle until it has four bytes in the buffer, then permits the process to continue. If a process got 30 bytes at once, then it would be able to read 7 integers successively without having to idle at all, then would need to idle for at least 2 more bytes. Also visible, above, is some logic forwarding any status messages outwards; that is a common idiom found in filters -- if they can't handle the event, they assume that something further along in the chain would. A filter is a process that consumes data from an input and yields results to an output, acting very much like a coroutine. Since they can be composed into chains, Wasp can make many tricky data flows easy to express and understand. For example, MOSREF's outbound cryptography chain:

;;; -- SNIP --
     (input-chain recv
                  (decrypt-filter recv-decrypt)
                  (check-collation-filter)
                  (check-checksum-filter crc32))

The first argument is the input that the next filter will wait on; each subsequent form spawns a new coroutine, supplying them with the supplied arguments for initialization. The result is a new input that provides a decrypted and carefully groomed sequence of messages from another MOSREF node. It's almost embarassing how simple that looks, like much of MOSREF's code. Much of my effort has been invested in Wasp Lisp for complicated I/O applications.

WaspVM Focus for 0.4..

In the key of Hubert J. Farnsworth, Good news, everyone! Wes Brown has offered to fund a port of MOSREF 2.0 to WaspVM, along with some other things. WaspVM 0.4 will contain the necessary primitive functionality to support MOSREF, and some modifications to the compiler to restore cross-platform compilation. After that, I will port MOSREF back to WaspVM, and maintain MOSREF until at least the release of Wasp Lisp 1.0. Wes has a couple more contributions to the project, but we need to hammer out the details before I make more announcements.

Wasp Lisp 0.3 on Mac OS X Fixed

There is a bug in LibEvent on Mac OS X when using any I/O scheduling method more sophisticated than select(2); Wasp Lisp 0.3.2 provides support for Mac OS X by disabling the more efficient kqueue(2) and poll(2) I/O schedulers on that platform and that platform only. It's disappointing, really, and any Mac OS X aficionado is more than welcome to find a more satisfactory way to achieve this -- I only use OS X when under duress. :)

Barely 24 Hours In..

.. and someone has already poked holes in the new I/O loop. I've cleaned up some issues related to the I/O loop, noninteractive programs, and some inconsistent I/O states that could crop up from exotic interleaving of the wait-input, send-output and exit primitives. Version 0.3.1 is now up on "Wasp Lisp Releases," with fixes for all the issues above and a wee bit more work on WaspDoc. Wes Brown has found some I/O problems on Mac OS X Leopard; I will dig out my old Mac OS X box and see what the issue is this evening or tomorrow. If it isn't just a compile problem, 0.3.2 may be following in the next 24 hours. Viva La Beta!

Ephemeral Security Found at an Archive Near You

Found this when digging for some old EphSec information for an email: Ephemeral Security: Mosquito Lisp Easy to Learn Looks like Archive.Org backed up the entire EphSec blog..