Overview

Overview

Genericore is a generic message queueing interface to provide an easily extensible stats processing infrastructure.

It is modelled after the EVA principle (Eingabe Verarbeitung Ausgabe). The system roughly contains three kinds of modules :

  • Input Modules (like E-MAIL gathering and simple parsing)
  • Processing Modules (like calculating the number of mails a user on the mailing list sends)
  • Output Modules (like a webserver which shows the calculated values)

The process we use is publish-subscribe (see below)

Message Queue

The message queue is an AMQP-based RabbitMQ which runs on:

  • 188.40.39.66,
  • default port,
  • defaul username and password (if you don't know it then simply ask someone at shackspace).

You might want to read Getting started with RabbitMQ (which is pretty good). Especially see Publish/Subscribe.

Two things are important:

  1. Queues: FIFO Pipelines which hold message
  2. Exchanges, which define how messages are emitted to queues

Default Values

Unless explicitly defined by the modules, the following default values are used:

1. Exchanges:

type = 'fanout'
passive = False
durable = False
auto_delete = False
internal = False
nowait = False

2. Queue:

name = ''
passive = False
durable = False
exclusive = True
auto_delete = True
nowait = False

Configure Rabbitmq

This is how the current rabbitmq server is configured to grant permission for user shack and do expose its interface to the network.

Download and install rabbitmq (there are plenty tutorials on the web)

In /etc/rabbitmq/rabbitmq.conf

NODENAME=rabbit                                                                    
NODE_IP_ADDRESS=YOUR.IP
NODE_PORT=5672

LOG_BASE=/var/log/rabbitmq
MNESIA_BASE=/var/lib/rabbitmq/mnesia

start the rabbitmq instance (e.g. /etc/rc.d/rabbitmq start), then change permissions

sudo rabbitmqctl delete_user guest
sudo rabbitmqctl add_user shack your_shack_password
sudo rabbitmqctl set_permissions shack ".*" ".*" ".*"

Current Modules

Currently the following modules are available:

Input Modules

genericore-ircspy

genericore-mailsrc

genericore-rabbithole

genericore-snmp_users

{ "type": "snmp", "subtype":0, "data": { TODO }}

Processing Modules

genericore-mail_proc

  • Description: Processes mail to generate statistics from new mails by saving and loading stats locally. Especially for given Users at the Mailing List
  • Stats generated:
    • Individual :
      • Number of Posts
      • First Mail date
      • Last Mail date
  • Maintainer: felix
  • Backend: mongodb
  • Input Exchange: mailsrc
  • Output Exchange: mail_proc
  • Output Protocol: see examples

genericore-mail_stats

Output Modules

genericore-bunghole

 { "type": "log", "subtype":0, "data": { TODO } } 

genericore-shiny

  • Description: (Currently not so) Shiny Web frontend for genericore (not yet plugable but functional prototype)
  • Maintainer: felix
  • Backend: cherrypy
  • Input Exchange: snmp_src,mail_stats,irc_something
  • Input Protocol
 Whatever the processing modules cough out 

genericore-irclog

Libraries

Javascript libraries

node-genericore

Python libraries

python-genericore

TODO Modules

describe the modules „todo“ in this section:

Input Modules

genericore-shackopen

  • Description : a simple source module which publishes the „Open“-Status of the Hackerspace (see wiki-status bar)

Processing Modules

genericore-mail_live_stats

  • Description: Generates live stats every N minutes from given input
  • Maintainer: felix
  • Stats generated:
    • Mails in the Hour,Day,Week
    • Most Active User in Hour,Day,Week
    • Biggest Thread in Hour,Day,Week
    • Biggest Mail in …
    • … EXTEND ME
  • Backend: mongodb
  • Input Exchange: mail_proc
  • Output Exchange: mail_live_stats (,log)
  • Output Protocol:
 { "type": "mail", "subtype":1, "data": { TODO } } 

Output Modules

genericore-milkshake

* Description: Gateway between AMQP and Websockets * Backend: Tornado,python-genericore * Maintainer: felix

project/genericore.txt · Zuletzt geändert: 2012/03/02 16:58 von 109.192.98.64