Cargo Cult Christianity

I talked with my father this morning, and struck upon an insight. In the aftermath of World War 2, the Allies left behind several airstrips in the South Pacific. The locals, who didn’t understand planes, saw these magic birds landing with massive loads of cargo during the war and when they took over the airstrip attempted to keep the cargo flying in.

But with no understanding of the process, all they could do was ape the mannerisms of the Allies they had observed. These cargo cults constructed coconut radios, rifles made of sticks, and even went onto the airstrip waving flags slavishly attempting to invoke the magic. We laugh at the idiosyncrasies of this.

Recently, I also came across a very insightful thought from James Burnham, from his book The Suicide of the West. I’ve reproduced it here.

We confront here a principle that would seem strangely paradoxical if it had not become so familiar in the thought and writings of our time. Liberalism is committed to the truth and to the belief that truth is what is discovered by reason and the sciences; and committed against the falsehoods and errors that are handed down by superstition, prejudice, custom and authority. But every man, according to liberalism, is entitled to his own opinion, and has the right to express it (and to advocate its acceptance). In motivating the theory and practice of free speech, liberalism must either abandon its belief in the superior social utility of truth, or maintain that we cannot be sure we know the truth. The first alternative—which would imply that error is sometimes more useful for society than the truth—is by no means self-evidently false, but is ruled out, or rather not even considered seriously, by liberalism. Therefore liberalism must accept the second alternative.

We thus face the following situation. Truth is our goal; but objective truth, if it exists at all, is unattainable; we cannot be sure even whether we are getting closer to it, because that estimate could not be made without an objective standard against which to measure the gap. Thus the goal we have postulated becomes meaningless, evaporates. Our original commitment to truth undergoes a subtle transformation, and becomes a commitment to the rational and scientific process itself: to—in John Dewey’s terminology—the “method of inquiry.”

How closely this parallels the cargo cults! We don’t know the truth, we cannot even know if we head in the right direction. Thus, the process takes the place of the thing itself. The parts are not greater than the whole.

I started this post off mentioning how I was struck by an insight this morning. Take these two concepts, Burnham’s process substituting for truth and cargo cults. Now apply them to the average church. When you walk in, it is just like those airfields. We’ve got waiting rooms, paying money, proselytizing, preaching, and singing. Yet it is hollow.

A pair of men walking down the road with none of those things can have more Christianity, the same way an open field or highway can act as more of an airfield than the mock field with flags being waved, waiting rooms and so on.

Cargo Cult Christianity

Marpa notes

This post gives a high level overview of the Marpa parser, with links to various resources on using it.

1957: Noam Chomsky publishes Syntactic Structures, one of the most influential books of all time. The orthodoxy in 1957 is structural linguistics which argues, with Sherlock Holmes, that “it is a capital mistake to theorize in advance of the facts”. Structuralists start with the utterances in a language, and build upward.

But Chomsky claims that without a theory there are no facts: there is only noise. The Chomskyan approach is to start with a grammar, and use the corpus of the language to check its accuracy. Chomsky’s approach will soon come to dominate linguistics.

What is marpa

Marpa is an Earley parser. You can find a lot of details on how it works internally from the author’s blog: Oceans of Awareness

How to use Marpa

Besides lots of snippet tutorials on the blog linked above, the Marpa docs are quite extensive and cover all of the options available. However, most users want some type of skeleton to get started with, and then start adding additional options.

This perl code gives a tiny skeleton to get started with, and which other options can be hooked in. Most specifically, take a look at the slif DSL documentation which defines what can be passed to the grammar in the source string.

use v5.16;
use strict;
use warnings;

use Marpa::R2;
use Data::Dump 'dd';
use Try::Tiny;

# create Marpa grammar
my $grammar = Marpa::R2::Scanless::G->new({
  source => \q{
    :discard ~ ws

    Sentence ::= WORD+ action => ::array
    WORD ~ 'foo':i | 'bar':i | 'baz':i | 'qux':i

    ws ~ [\s]+

# create recognizer to use the grammer and "drive" parsing
my $recce = Marpa::R2::Scanless::R->new({ grammar => $grammar });

# get the input to send in from somewhere
my $input = '1) bargain bebar Foo bar: baz and qux, therefore qux (foo!) implies bar.';

# try and read the input
try { $recce->read(\$input) };
# have we gotten to the end of the input yet?
while ($recce->pos < length $input) {
  # if not:
  try   {       $recce->resume                    }  # restart at current position
  catch { try { $recce->resume($recce->pos + 1) } }; # advance the position by one character
  # if both fail, we go into a new iteration of the loop.

# process the parse to produce the value
dd $recce->value;


Ruby slippers technique

Once a grammer has been created, the recognizer is used to “drive” it over the input string. Marpa is more powerful than other parsing technologies, and allows for arbitrary pausing during this process. One option that can be given to the recognizer is the rejection flag, which defines whether marpa throws an exception when it cannot parse, or produces an event and pauses. This provides a powerful interface for getting over messy input that does not fit with the proper grammar.

The following code is more complicated, but demonstrates using this technique to add pretend values whenever there is missing input to fix up a bad input. The input string given is missing ; characters at various points that the grammar expects. When this happens, Marpa pauses with a ‘rejection event. Note the leading single quote. Once paused, the ruby slippers function calls various recognizer functions to check on the current state of Marpa, to see what’s going on. If the next expected value is the ; and nothing else, the function tells the recognizer it just got one, and restarts parsing.


use v5.16;
use strict;
use warnings;

use Marpa::R2;
use Data::Dump 'dd';

my $grammar = Marpa::R2::Scanless::G->new({
  source => \q{
    :discard ~ ws

    Block         ::= Statement+ action => ::array
    Statement     ::= StatementBody (STATEMENT_TERMINATOR) action => ::first

    StatementBody ::= 'statement'       action => ::first
                    |   ('{') Block ('}') action => ::first


    #event ruby_slippers = predicted STATEMENT_TERMINATOR

    ws ~ [\s]+


my $recce = Marpa::R2::Scanless::R->new({
  grammar => $grammar,
  trace_terminals => 1,
  trace_values => 1,
  rejection => "event"

my $input = q(
  { statement }

# this is copy and pasted :D
for (
  $recce->pos < length $input;
) {
  for my $event ( @{ $recce->events() } ) {
    my ($name) = @{$event};
    if ($name eq "'rejected") {
      ruby_slippers($recce, \$input);

# we've exhausted all input, so check if we need a final terminator
ruby_slippers($recce, \$input);

# now give us the parse!
dd $recce->value;

sub ruby_slippers {
  my ($recce, $input) = @_;
  my %possible_tokens_by_length;
  my @expected = @{ $recce->terminals_expected };
  for my $token (@expected) {
    if ($token eq 'STATEMENT_TERMINATOR' && $#expected+1 == 1) {
      push @{ $possible_tokens_by_length{0} }, [STATEMENT_TERMINATOR => ';'];

  my $max_length = 0;
  for (keys %possible_tokens_by_length) {
    $max_length = $_ if $_ > $max_length;

  if (my $longest_tokens = $possible_tokens_by_length{$max_length}) {
    for my $lexeme (@$longest_tokens) {
    $recce->lexeme_complete($recce->pos, $max_length);



Useful links

Marpa resources

Finding a pattern in a chunk of text

Parsing timeline

Code examples originally from

Super fancy print all parse trees

Discard AND use whitespace

Marpa notes

Testing framework

My coworkers have put together a testing framework which does some pretty fancy things. It’s able to record test runs against live systems, and then we can check in the results to use for future tests. That lets them work around not having development servers for a variety of different services. Record a run against prod, and save that for future tests.

However, to do this they need to send in a custom data format into their wrapper telling it what to do. The wrapper runs a series of functions to actually build and run the tests they want. Today I got to look at how this wrapper works and they’ve built a state machine to implement recursive descent parsing! So often developers don’t realize they are creating a parser, and don’t think about the tooling as building a language for the problem space.

So instead, we’ve got the raw input data structure, acting as a quasi pre-parsed AST to recurse down. Of course, you have to check the contents of various keys each step of the way to decide which function to run next, and what input keys to grab. And if you don’t have the keys? Boom, return an error for what’s expected “next”. Almost like expecting a noun before a verb in a sentence, with both required.

So none of this code is easily modified, nor is the resulting language obvious. And the only benefit they really get is that they can skip lexing a string and just feed in a raw data structure that…they…wrote… manually. Sometimes this job makes me sad.

Testing framework


Recently, I got the team at work to start using Ansible for managing a couple of servers. Sadly, they were in such a tearing hurry in the later weeks that they didn’t bother to use the tool. Ah well, here’s the notes I put together for them after I spent an hour learning how to use it.

Ansible doesn’t require much to get started. Just install it on the system you plan to use for managing boxes. Note that each user *could* install their own copy locally, but would then not share useful info about the systems to be managed without some synchronization of a few files. These docs cover how to install ansible in a variety of ways. Easiest is just using the linux server’s package manager. Then you can immediately use adhoc commands against multiple systems.

Of course, most times you want a written down set of repeatable instructions. Ansible uses playbooks for that. The intro quickly goes over the basics, and the examples allows you to poke through nicer laid out playbooks. They have proper organization, linking to subcomponents, and managing multiple systems, such as in the lamp example provided.

However, there isn’t always a need for a large and complex system such as that when all you want to do is set up user accounts. To create a playbook capable of that, follow these instructions.

  1. mkdir playing-with-users; cd playing-with-users@
  2. touch site.yml hosts
  3. vim hosts and add content
  4. vim site.yml and add content
  5. ansible-playbook -i hosts site.yml




# This playbook configures the list of users for all servers
      shell: bash
      shell: bash
      shell: fish

  - name: Install packages onto server
    package: name={{ item }} state=present
     - git
     - fish

  - name: create users on all nodes
    hosts: servers
    remote_user: root

    # Add each user with a bash shell, appending the group 'admins' and 'developers' to the user's groups
    # see the user docs for options like ssh key config
    user: name={{ item.key }} shell=/bin/{{ }} groups=admins,developers append=yes
    with_dict: {{ users }}

The core modules like the user or package modules are idempotent, and can be run as many times as you want. They will only make the changes necessary on the system. So you can run this playbook, then add more users and run it again. Only the new user will be configured.


User module docs

List of modules available


Complex systems

Any system that actually does useful work tends towards the complex. There are far more edge cases and complexities to real life than a cursory look shows. This is one reason that you shouldn’t roll your own libraries for everything. Not only do you have an infinitely regressing problem (do you build compilers for your custom language?), but you also simply don’t have the details.

Until you muck about in a problem space, you don’t know what you don’t know. You already have your own problem to solve. The problem may include writing your own library to support it. If so, that will become readily apparent. Don’t prematurely optimize complexity. You’ll have more than enough to deal with soon.

Complex systems

Joining a project

One of the primary reasons it takes at least three months to ramp up on a project is simply due to communication. The vast majority of the meta information about the project lives in the heads of the developers. Even if they have all the code cleanly laid out, in one repository, with lots of comments, it will still be hard to sync mentally.

  • Why was X done instead of Y
  • Where are we going
  • Where did we come from
  • What are the standards

Anything beyond a trivial code base has these issues. Since it only lives in the devs heads, it takes time for them to remember, and then to tell you. So you accumulate bits until all is clear.

Joining a project

A functional local build

So I’ve got the python marpa bindings pulled out and working locally. Packaging this up is not quite as obvious as I would have expected. You’d think there’d be instructions for how to build a c lib/install a needed library as part of or something.

Found where marpa hid the built library! Now I just need to figure out what bits I need for cffi. I can have the tarball in the repo, unpack it on target system, make, then tell to install the shared lib.

A functional local build