init

wcmtools/perlbal/doc/Classes.txt

@@ -0,0 +1,72 @@
+Perlbal Classes -- a brief introduction/overview.
+
+Socket -- descends from Danga::Socket
+  Adds on to the base class to provide some functionality specifically
+useful for creating HTTP sockets.
+
+TCPListener -- descends from Perlbal::Socket
+  Very lightweight and fast connection accept class.  Takes incoming
+connections as fast as possible and passes them off, instantiating one of
+the various Client* classes to handle it.
+
+BackendHTTP -- descends from Perlbal::Socket
+  This class handles connections to the backend web nodes for getting data
+back to the user.  This class is used by other classes such as ClientProxy
+to send a request to an internal node.
+
+HTTPHeaders --
+  Header management.  Parses headers (request and response) and stores data
+for further user.  Also manages validation of the request line so that it
+conforms to HTTP specifications.
+
+ClientHTTPBase -- descends from Perlbal::Socket
+  Provides base functionality to ClientHTTP and ClientProxy.  Notably, the
+ability to efficiently send files to the remote user.  Also handles most of
+the state logic for statistics and such.
+
+ClientHTTP -- descends from Perlbal::ClientHTTPBase
+  Very simple and lightweight class.  Handles sending files to the user
+without much overhead.  Most of the functionality is contained in the parent
+class, and this class doesn't implement much new stuff.
+
+ClientProxy -- descends from Perlbal::ClientHTTPBase
+  Takes an incoming connection from a user and connects to a backend node
+(BackendHTTP) and relays the request.  The backend can then either tell the
+proxy to reproxy and load a file from disk, or return a file directly, or
+just return a status message.
+
+ClientManage -- descends from Perlbal::Socket
+  Simple interface that provides a way for users to use the management
+interface of Perlbal.  You can connect to the management port (as defined
+in the config file) with a web browser or regular telnet.
+
+Service --
+  A service is a particular item that Perlbal is doing.  Services can have
+a role which defines how they behave.  Each service can also have a bunch
+of parameters set to further adjust its behavior.  By itself, the Service
+class handles maintaining pools of backend connections and managing statistics
+about itself.
+
+StatsListener -- descends from Perlbal::Socket
+  This class listens for UDP broadcasts from the web nodes describing how
+many available children they have.  This information is then used to pick an
+endpoint for a backend connection to be made to in order to handle a user's
+incoming request.
+
+
+
+                            {{ INTERNET }}
+                                  |
+                                  v
+              [Service]<===>[TCPListener]
+                          ___/    |    \___
+                         v        v        v
+             [ClientManage]  [ClientHTTP] [ClientProxy]
+                                                ^
+                                                |
+                                                v
+                                          [BackendHTTP]
+
+So connections come in from wherever and get to the TCPListener.  It uses
+Service objects to determine what kind of Client* to spawn.  The Client
+classes then handle crafting the response for the user.

wcmtools/perlbal/doc/Hooks.txt

@@ -0,0 +1,106 @@
+Perlbal Hooks --
+
+                            -- INTRODUCTION --
+
+Basically, a hook is a bit of code that is run at certain stages in the
+requests that Perlbal handles.  There are all kinds of hooks available and
+they all do different things.  Some are only applicable to some of the
+roles and others are applicable only to certain classes.  Each hook is
+described in detail below, but first a description of the basics of a hook.
+
+In general, you define a hook by calling the "register_hook" method on a
+Perlbal::Service object.  You specify what hook you are interested in and
+provide a reference to a subroutine that will be called with the parameters
+particular to that hook.
+
+There are three types of hooks:
+
+--- Global hooks
+These are hooks that are defined on a global scale.  They are set like so:
+
+    Perlbal::register_global_hook('foo', sub { return 0; });
+
+That would define a global hook named foo that would return 0 when it's
+called.  (Return codes from hooks will be explained below.)
+
+--- Service handler hooks
+A handler hook is attached to a particular service.  These hooks are called
+one at a time, starting from the top of the hook list on a service, until
+one hook returns 1.  At that point, no further hooks are called.  For
+example:
+
+    $service->register_hook('bar', sub {
+        # do something
+        return 1;
+    });
+
+When this hook runs, it would return 1, signalling to Perlbal that it had
+done what it needed to do and that Perlbal shouldn't call any further
+hooks.  You can use this type of hook to create sets of plugins that all
+handle different types of requests, and when one hook had handled a request
+it wouldn't continue telling other hooks about the request.
+
+--- Service general hooks
+These hooks are defined the same way as above, but general hooks are all
+run.  The return code is ignored.  This can be useful for putting in code
+that records statistics about an action or something to that effect.
+
+
+                               -- HOOKS --
+
+The following hooks are defined in the Perlbal distribution:
+
+GENERAL     end_proxy_request           Perlbal::ClientProxy
+This hook is called when the ClientProxy object is being closed.
+
+HANDLER     start_proxy_request         Perlbal::ClientProxy
+Called as soon as we've read in headers from a user but right before we've
+requested a backend connection.  If a true value is returned, Perlbal will
+not request a backend.
+
+HANDLER     start_file_reproxy          Perlbal::ClientProxy, $filename_ref
+Called when we've been told to reproxy a file.  If you return a true
+value, Perlbal will not perform any operations on the file and will simply
+return.  You can also change the file in the scalar ref passed as the
+second parameter.
+
+HANDLER     backend_client_assigned     Perlbal::BackendHTTP
+Happens right after a backend is given a client, but before we've talked to
+the backend and asked it to do something.  If you return a true value, the
+process is stopped and you will manually have to send the client's request
+to the backend, etc.
+
+HANDLER     start_web_request           Perlbal::ClientHTTP
+When a 'web' service has gotten headers and is about to serve it... return
+a true value to cancel the default handling of web requests.
+
+HANDLER     start_send_file             Perlbal::ClientHTTPBase
+Called when we've opened a file and are about to start sending it to the
+user using sendfile.  Return a true value to cancel the default sending.
+
+HANDLER     start_serve_request         Perlbal::ClientHTTPBase, $uri_ref
+Called when we're about to serve a local file, before we've done any
+work.  You can change the file served by modifying $uri_ref, and cancel the
+process by returning a true value.
+
+HANDLER     backend_response_received   Perlbal::BackendHTTP
+Called as soon as response headers are read from the backend.  If you
+return a true value, will stop all handling at that point.
+
+HANDLER     handle_put                  Perlbal::ClientHTTP
+Called whenever we have data to write to a file being PUT.  Return a true
+value to cause Perlbal to not actually save the file or do anything with
+it.
+
+HANDLER     setup_put                   Perlbal::ClientHTTP
+Called as soon as we've gotten headers requesting a PUT.  Cancels the
+default work if you return a true value.  Careful: this function sets up
+a file descriptor for use in writes in handle_put, so if you're going to
+override setup_put you will probably want to override handle_put.
+
+HANDLER     make_high_priority          Perlbal::ClientProxy
+Called when a request is received and right before we're about to determine
+if this request is high priority or not.  Return a true value to make the
+request high priority; false to leave it alone.  Note that this is only
+called when the request isn't already high priority due to cookie priority
+scheduling, which is done inside Perlbal's Service module.

wcmtools/perlbal/doc/README.txt

@@ -0,0 +1,10 @@
+Perlbal is not yet documented.
+
+See the website, conf/ directory for examples, and post questions on
+the mailing list for now.
+
+Website:
+http://www.danga.com/perlbal/
+
+Mailing list:
+http://lists.danga.com/mailman/listinfo/perlbal

wcmtools/perlbal/doc/Reproxying.txt

@@ -0,0 +1,47 @@
+Perlbal supports the concept of reproxying.  Basically, this gives it the
+ability to ask a backend node for a file and get back a specific header
+that says "this file is really over there, get it there."  Perlbal will
+then load that file or URL and send it to the user transparently, without
+them ever knowing that they got reproxied to another location.
+
+This can be useful for having URLs that get mapped to files on disk without
+giving users enough information to map out your directory structure.  For
+example, you can create a file structure such as:
+
+/home/pics/$userid/$pic
+
+Then you can have URLs such as:
+
+http://foo.com/mysite/users/$userid/picture/$pic
+
+When this URL gets passed to the backend web node, it could return a simple
+response that includes this header:
+
+X-REPROXY-FILE: /home/pics/$userid/$pic
+
+Perlbal will then use asynchronous IO to send the file to the user without
+slowing down Perlbal at all.
+
+This support also extens to URLs that can be located anywhere Perlbal has
+access to.  It's the same syntax, nearly:
+
+X-REPROXY-URL: http://foo.com:80/resource.html
+
+You can also specify multiple URLs:
+
+X-REPROXY-URL: http://foo.com:80/resource.html http://baz.com:8080/res.htm
+
+Just specify any number of space separated URLs.  Perlbal will request them
+one by one until one returns a response code of 200.  At that point Perlbal
+will proxy the response back to the user just like normal.
+
+Note that the user's headers are NOT passed through to the web server.  To
+the target server, it looks simply like Perlbal is requesting the resource
+for itself.  This behavior may change at some point.
+
+One final note: the server that returns the reproxy header can also return
+a X-REPROXY-EXPECTED-SIZE header.  If present, Perlbal will consider
+a reproxy a failure if the file returned by the target system is of a
+different size than what the expected size header says.  On failure,
+Perlbal tries the next URI in the list.  If it's a file being reproxied, a
+404 is returned if the file size is different.

wcmtools/perlbal/doc/todo.txt

@@ -0,0 +1,29 @@
+* Keep track of how many times we couldn't do keep-alive from the backend.  This
+  happens if the backend server is doing chunked encoding or doesn't provide a
+  content length.
+
+* Allow option to say O_EXCL for PUTs
+
+* Investigate/fix error with "no mapping for fd" when a socket abruptly closes
+  during the connect process.  (Run ab against perlbal, then Ctrl+C it.)
+
+* Investigate when BackendHTTP fails on a call to get_res_headers?
+
+* Fix shutdown graceful to close off persistent clients and bored backends (reproxy too?)
+
+* Document the 'reproxy bored' queue [old...new] self cleaning on the old side, but
+  uses the new side when it needs a connection
+
+* Investigate why Perlbal is slow to accept connections
+
+* Lots of work can be done to reduce system calls:
+  - coalesce epoll_ctl calls
+  - make socket by hand without IO::Socket::INET
+  - edge-triggered epoll in Danga::Socket
+  - don't cork twice on reproxy & send
+
+* Support dictionary (scratch hash) per service
+  - can configure keys to send to backends
+  - backends can set keys in dictionary (so they propogate to other backends, etc)
+
+* Allow configurable response code that means "backend dead; replay request"