Re: software end of conspiracy

On Sat, Jan 16, 2010 at 4:57 PM, Barrett Brown <barriticus@gmail.com> wrote:

I probably mentioned that Charles Johnson has agreed to work with us on establishing a sort of next-generation networking software/paradigm that will be used to better inform the populace on the failures of various media figures. Andrew Stein has just finished the basic schematics/tech outline of the core software. Just skim it, I don't understand most of it either; my role is more along the lines of recruitment, organization, and promotion, since we'll need a group of bloggers to agree to use this. Stein will be approaching some investors for seed capital soon; they asked him to bring them a project sort of along these lines a few months ago, and with Johnson and I attached to the project, it's all the more desirable. Sorry if this all sounds vague; we're sort of in the midst of solidifying a great deal of this. Of course, this is all secret for now.

Ego

Distributed Social Networking for John Q. Developer

ABSTRACT

______________________________________________________________________

Ego is a Content Management System built on an XMPP based adhoc social

network. The product itself consists of an XMPP server written in

Clojure, and a selection of HTML/JQuery widgets that implement simple,

embeddable, stylable functionality without templating - contact lists,

messaging, profile browsing, etc.

Adhoc

Ego is a social network in the truest sense - there is no hub, no root

server and no central authority. Like email (and XMPP, on which the

federation protocol is implemented), users on the Ego network are

identified by a username and domain, not simply username - Andrew is

now Andrew@ego.fm. By abstracting identity to the domain level,

identity itself becomes domain independent, and common facebook style

social networking functionality becomes "adhoc" - Andrew@ego.fm and

Josh@analgoatsex.com can freely communicate as if they were on the

same site, despite Josh's apparent obsession with deviant sexual

fetishism.

Whitelabel

Ego comes unstyled and without any preconceived notions of the

developer's content schema - it simply attaches common social

networking functionality onto an existing application or design,

through the adhoc network. Brands, groups and content providers can

build there own site on the Ego platform and leverage this

functionality across the adhoc network.

Simple

Ego is designed to be as flexible as possible with regards to design -

the backend is totally configurable, the front end is untemplated, and

core extensions can be directly integrated in Java, Scala, Clojure,

JRuby, Jython or Groovy - plus, there is nothing preventing developers

from simply building around core's functionality in any other language

...

INSTALLATION

______________________________________________________________________

Ego is built on Clojure, JDK 6+ and your choice of Backend (Redis and

Postgresql currently come built in). Step one should be to configure

your application. Ego's configuration is written in pure clojure

(log4j aside ... TODO), and is contained in the config.clj and

sql/{backend-of-choice}.clj files. By default, Ego will try to use a

Postgresql instance with username "postgresql" and password

"password", and will serve a self-signed SSL certificate passworded

with "password" - you probably don't want these things configured as

such, unless you are just hacking around.

Once you have everything configured as you want it, install Leiningen

(http://www.github.com/technomancy/leiningen) and get started:

git clone http://www.github.com/texodus/ego

cd ego

lein deps

lein compile

lein-repl

... should bring you up a repl. From there, you'll find the following

commands useful:

;; Start the server

(org.ego/-main)

;; Initialize a blank schema

(org.ego.core.accounts/setup)

;; Create a new user

(org.ego.core.accounts/create-user "username" "password")

Configuration settings can be found in config.clj, and database

settings can be found in sql/{backend type}.clj

ARCHITECTURE

______________________________________________________________________

From a simplistic standpoint, the Ego platform can be viewed as an

XMPP server with a few custom XEPs, and a custom, modular web client

that is aware of these XEPs. It should thus come as no surprise that

the Ego Network infrastructure is identical to the XMPP federation

protocol architecture (because that's what it is):

User <--JSON--> Node <------XMPP------> Node <--JSON--> User

^ ^

| |

--XMPP--> Node <--XMPP--

^

|

--JSON--> User

On the User side, the user requests the widget HTML and runs the

embedded JQuery - from there, the widget maintains an HTTP connection

with the server and passes data back and forth in JSON format. The

current implementation is via HTTP long polling - this should probably

be refactored to BOSH, though this will substantially increase the

complexity of the widgets (as they will have to implement a

substantial subset of the XMPP client protocol, as well as share a

good deal of message routing logic where they are currently

independent).

On the server side, each node opens and maintains a (timeout and

max-connections restricted) XMPP channel with each additional node as

it queues messages for those nodes, OR it accepts incoming connections

from other nodes and replies to requests about its's local content and

user "state."

Each node consists of these two transports (JSON and XMPP), a core

module responsible for routing and user "state," and a datastore:

**********************

* Node --------------> DB

* | *

* v *

* ---> Core <--- *

* | | *

* v v *

* JSON XMPP *

*** ^ ********** ^ ***

| |

-> Internet <-

Core is responsible for maintaining user and (internal) content state.

The JSON and XMPP modules do not make routing decisions on their own,

nor do they request content directly from the datastore - instead,

these actions are encoded as messages to Core, which may then request

data from the datastore, request new client channels from XMPP or

simply reinject the content into the proper channel.

The datastore itself need be as flexible as possible to allow

arbitrary development against potentially pre-existing (and

potentially of nontrivial complexity) aplpication stacks. JDBC allows

use to write CRUD functionality using a simply query flatfile.

*****************

* Core *

----------- -----------

| * | | * |

JSON * v v * XMPP

^ * Queue * ^

| * ^ * |

| * | * |

| * v * |

-------- Routing --------

* ^ *

********|********

|

--> JDBC <----> DB

^

|

---- (load-file "some-sql.clj")

The XMPP module actually consists of two seperate stacks: a server

stack and a client stack, both of which end in the Jabber namespace,

where the actual XMPP processing logic is implemented. This

architecture mirrors Netty's Pipeline hierarchy, and is nonblocking.

Each upstream request from either a client or server Channel is pushed

up the Netty Pipeline stack, and results in a Message being queued in

Core for routing. Core then emits Messages to either the client or

Server write function wrappers, which results in the Messages being

pushed downstream the appropriate channel.

------------ -> Core <- ------

| | | | | |

| *********v****|********|****v*** |

| * Jabber X X X X * |

| *********|****^********^****|*** |

| | | | | |

| *********v****|********|****v*** | X = Processing Logic

| * Stanza X X X X * |

| *********|****^********^****|*** |

| | | | | |

| *********v****|********|****v*** |

| * XML X X X X * |

| *********|****^********^****|*** |

| | | | | |

**|**********v****|** **|****v****|*********

* X Client X X * * X X X Server *

**^**********|****^** **^****|****^*********

| | | | | |

| ------Internet------ |

| |

------------------ Core ------------

The JSON module is a good deal simpler, as its is based on

pre-existing abstractions provided by the Java platform. Current

architecture relies on Compojure as a servlet abstraction, with one

servlet dedicated to serving JSON messages for each widget, with

servlet URLs unique to each widget. Once a JSON message has been

received, it is processed in teh servlet and queued into Core - when

Core later emits a Message for a JSON client, the message is queued in

the servlet until the next client poll.