zlax/ljr
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

init

Browse Source
This commit is contained in:
ivan
2019-02-06 00:49:12 +03:00
commit 8dbb1bb605
4796 changed files with 506072 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
wcmtools/memcached/website/_config.bml Executable file
View File

@@ -0,0 +1,2 @@
LookRoot .
ForceScheme memcached

78
wcmtools/memcached/website/apis.bml Executable file
View File

@@ -0,0 +1,78 @@
<?page
wintitle=>client apis
body<=
<?h1 Perl API h1?>
<p> An object-oriented Perl module can be found on CPAN as <tt>Cache::Memcached</tt> or downloaded here.</p>
<ul>
<li><a href="dist/Cache-Memcached-1.14.tar.gz">Cache-Memcached-1.14.tar.gz</a>, GPL/Artistic.
[<a href="http://cvs.danga.com/browse.cgi/wcmtools/memcached/api/perl/ChangeLog?rev=HEAD&amp;content-type=text/plain">ChangeLog</a>]
</ul>
<p>The API takes advantage of the server's opaque flag support and
sets its "complex" flag whenever the object being stored or retrieved
isn't a plain scalar. In that case, the <tt>Storable</tt> module is
used to freeze and thaw the value automatically going in and out of
the memcached.</p>
<?h1 PHP API h1?>
<p>Several PHP APIs are available:</p>
<ul>
<li><a href="http://phpca.cytherianage.net/memcached/">http://phpca.cytherianage.net/memcached/</a></li>
<li><a href="http://pecl.php.net/package/memcache/">http://pecl.php.net/package/memcache/</a> -- implemented in C, but doesn't support multiple memcached servers</li>
<li>... (many others, send me links?)</li>
</ul>
<?h1 Python API h1?>
<p>An object-oriented Python module modelled after the Perl one is also
included in the distribution. It includes Pickle/cPickle support (like Perl's Storable).</p>
<ul>
<li><a href="dist/python-memcached-1.2.tar.gz">python-memcached-1.2.tar.gz</a>, Python license.
[<a href="http://cvs.danga.com/browse.cgi/wcmtools/memcached/api/python/ChangeLog?rev=HEAD&amp;content-type=text/plain">ChangeLog</a>]
</ul>
<p>The Python API, while tested, has not experienced real-world production use.
Also, its performance could be optimized (less syscalls parsing responses), if anybody
out there would care to improve it.</p>
<p>Please contact us to take over its development.</p>
<?h1 Ruby API h1?>
<p>A Ruby API is available from Michael Granger at:</p>
<ul>
<li><a href="http://www.deveiate.org/code/Ruby-MemCache.html">http://www.deveiate.org/code/Ruby-MemCache.html</a></li>
</ul>
<?h1 Java API h1?>
<p>A Java API is maintained by <a href="mailto:greg@meetup.com">Greg Whalin</a> from <a href="http://www.meetup.com/">Meetup.com</a>. You can find that library here:
<ul>
<li><a href="http://www.whalin.com/memcached/">http://www.whalin.com/memcached/</a> -- Java API for memcached
</ul>
<?h1 C API h1?>
<p>Multiple C libraries for memcached exist:</p>
<ul>
<li><a href="http://www.outoforder.cc/projects/libs/apr_memcache/">apr_memcache</a> by Paul Querna; Apache Software License version 2.0</li>
<li><a href="http://people.freebsd.org/~seanc/libmemcache/">libmemcache</a> by Sean Chittenden; BSD license.</li>
</ul>
<?h1 Protocol h1?> <p>To write a new client, check out the <a
href="http://cvs.danga.com/browse.cgi/wcmtools/memcached/doc/protocol.txt?rev=HEAD&amp;content-type=text/plain">protocol
docs</a>. Be aware that the most important part of the client is the
hashing across multiple servers, based on the key, or an optional
caller-provided hashing value. Feel free to join the mailing list (or
mail me directly) for help, inclusion in CVS, and/or a link to your
client from this site.</p>
<=body
page?>

18
wcmtools/memcached/website/download.bml Executable file
View File

@@ -0,0 +1,18 @@
<?page
wintitle=>download
body<=
<?h1 Download Server h1?>
<p>The server is licensed under the BSD License. Read the <a href="http://cvs.danga.com/browse.cgi/wcmtools/memcached/ChangeLog?rev=HEAD&amp;content-type=text/plain">ChangeLog</a> for details on what's new.</p>
<ul>
<li><a href="dist/memcached-1.1.12.tar.gz">memcached-1.1.12.tar.gz</a> -- Apr 4, 2005
</ul>
<?h1 Download Client APIs h1?>
<p>The client APIs, licensed individually, are available for download on the <a href="apis.bml">Client APIs</a> page.</p>
<=body
page?>

116
wcmtools/memcached/website/index.bml Executable file
View File

@@ -0,0 +1,116 @@
<?page
wintitle=>a distributed memory object caching system
body<=
<?h1 What is <?memd?>? h1?>
<p><?memd?> is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.</p>
<p><a href="http://www.danga.com/">Danga Interactive</a> developed <?memd?> to enhance the speed of <a href="http://www.livejournal.com/">LiveJournal.com</a>, a site which was already doing 20 million+ dynamic page views per day for 1 million users with a bunch of webservers and a bunch of database servers. <?memd?> dropped the database load to almost nothing, yielding faster page load times for users, better resource utilization, and faster access to the databases on a memcache miss.</p>
<?h1 How it Works h1?>
<p>First, you start up the <?memd?> daemon on as many spare machines as you have. The daemon has no configuration file, just a few command line options, only 3 or 4 of which you'll likely use:
<pre class='example'># ./memcached -d -m 2048 -l 10.0.0.40 -p 11211</pre>
<p>This starts <?memd?> up as a daemon, using 2GB of memory, and listening on IP 10.0.0.40, port 11211. Because a 32-bit process can only address 4GB of virtual memory (usually significantly less, depending on your operating system), if you have a 32-bit server with 4-64GB of memory using PAE you can just run multiple processes on the machine, each using 2 or 3GB of memory.</p>
<?h1 Porting the Application h1?>
<p>Now, in your application, wherever you go to do a database query, first check the memcache. If the memcache returns an undefined object, then go to the database, get what you're looking for, and put it in the memcache:</p>
<pre class='example'>
<div class='exampletitle'>Perl Example (see <a href="apis.bml">APIs page</a>)</div>
sub get_foo_object {
my $foo_id = int(shift);
my $obj = $::MemCache->get("foo:$foo_id");
return $obj if $obj;
$obj = $::db->selectrow_hashref("SELECT .... FROM foo f, bar b ".
"WHERE ... AND f.fooid=$foo_id");
$::MemCache->set("foo:$foo_id", $obj);
return $obj;
}</pre>
<p>(If your internal API was already clean enough, you should only have to do this in a few spots. Start with the queries that kill your database the most, then move to doing as much as possible.)</p>
<p>You'll notice the data structure the server provides is just a dictionary. You assign values to keys, and you request values from keys.</p>
<p>Now, what actually happens is that the API hashes your key to a unique server. (You define all the available servers and their weightings when initializing the API) Alternatively, the APIs also let you provide your own hash value. A good hash value for user-related data is the user's ID number. Then, the API maps that hash value onto a server (modulus number of server buckets, one bucket for each server IP/port, but some can be weighted heigher if they have more memory available).</p>
<p>If a host goes down, the API re-maps that dead host's requests onto the servers that are available.</p>
<?h1 Shouldn't the database do this? h1?>
<p>Regardless of what database you use (MS-SQL, Oracle, Postgres, MysQL-InnoDB, etc..), there's a lot of overhead in implementing <a href="http://www.wikipedia.org/wiki/ACID">ACID</a> properties in a RDBMS, especially when disks are involved, which means queries are going to block. For databases that aren't ACID-compliant (like MySQL-MyISAM), that overhead doesn't exist, but reading threads block on the writing threads.</p>
<p><?memd?> never blocks. See the "Is memcached fast?" question below.</p>
<?h1 What about shared memory? h1?>
<p>The first thing people generally do is cache things within their
web processes. But this means your cache is duplicated multiple
times, once for each mod_perl/PHP/etc thread. This is a waste of
memory and you'll get low cache hit rates. If you're using a
multi-threaded language or a shared memory API (IPC::Shareable, etc),
you can have a global cache for all threads, but it's per-machine. It doesn't scale to multiple machines.
Once you have 20 webservers, those 20 independent caches start to look
just as silly as when you had 20 threads with their own caches on a
single box. (plus, shared memory is typically laden with limitations)</p>
<p>The <?memd?> server and clients work together to implement one
global cache across as many machines as you have. In fact, it's
recommended you run both web nodes (which are typically memory-lite
and CPU-hungry) and memcached processes (which are memory-hungry and
CPU-lite) on the same machines. This way you'll save network
ports.</p>
<?h1 What about MySQL 4.x query caching? h1?>
<p>MySQL query caching is less than ideal, for a number of reasons:</p>
<ul class='spaced'>
<li>MySQL's query cache destroys the entire cache for a given table whenever that table is changed. On a high-traffic site with updates happening many times per second, this makes the the cache practically worthless. In fact, it's often harmful to have it on, since there's a overhead to maintain the cache.</li>
<li>On 32-bit architectures, the entire server (including the query cache) is limited to a 4 GB virtual address space. <?memd?> lets you run as many processes as you want, so you have no limit on memory cache size.</li>
<li>MySQL has a query cache, not an object cache. If your objects require extra expensive construction after the data retrieval step, MySQL's query cache can't help you there.</li>
</ul>
<p>If the data you need to cache is small and you do infrequent updates, MySQL's query caching should work for you. If not, use <?memd?>.</p>
<?h1 What about database replication? h1?>
<p>You can spread your reads with replication, and that helps a lot,
but you can't spread writes (they have to process on all machines) and
they'll eventually consume all your resources. You'll find yourself
adding replicated slaves at an ever-increasing rate to make up for the
diminishing returns each addition slave provides.</p>
<p>The next logical step is to horizontally partition your dataset
onto different master/slave clusters so you can spread your writes,
and then teach your application to connect to the correct cluster
depending on the data it needs.</p>
<p>While this strategy works, and is recommended, more databases (each
with a bunch of disks) statistically leads to more frequent hardware
failures, which are annoying.</p>
<p>With <?memd?> you can reduce your database reads to a mere
fraction, leaving the databases to mainly do infrequent writes, and
end up getting much more bang for your buck, since your databases
won't be blocking themselves doing ACID bookkeeping or waiting on
writing threads.</p>
<?h1 Is <?memd?> fast? h1?> <p>Very fast. It uses <a
href="http://www.monkey.org/~provos/libevent/">libevent</a> to scale
to any number of open connections</a> (using <a
href="http://www.xmailserver.org/linux-patches/nio-improve.html">epoll</a>
on Linux, if available at runtime), uses non-blocking network I/O, refcounts internal objects
(so objects can be in multiple states to multiple clients), and uses
its own slab allocator and hash table so virtual memory never gets
externally fragmented and allocations are guaranteed O(1).</p>
<?h1 What about race conditions? h1?>
<p>You might wonder: <i>"What if the <tt>get_foo()</tt> function adds a stale version of the Foo object to the cache right as/after the user updates their Foo object via update_foo()?"</i></p>
<p>While the server and API only have one way to get data from the cache, there exists 3 ways to put data in:</p>
<ul class='spaced'>
<li><b>set</b> -- unconditionally sets a given key with a given value (<tt>update_foo()</tt> should use this)</li>
<li><b>add</b> -- adds to the cache, only if it doesn't already exist (<tt>get_foo()</tt> should use this)</li>
<li><b>replace</b> -- sets in the cache only if the key already exists (not as useful, only for completeness)</li>
</ul>
Additionally, all three support an expiration time.
<=body
page?>

59
wcmtools/memcached/website/memcached.css Executable file
View File

@@ -0,0 +1,59 @@
body {
font-family: sans-serif;
border: 5px solid #7070d0; margin: 0; padding: 30px;
background: white;
}
a { color: #5050d0; }
h1 {
color: #7070d0;
letter-spacing: 0.2em;
text-align: center;
width: auto;
font-size: 20pt;
margin-bottom: 0;
}
h2 {
font-size: 13pt;
font-style: italic;
margin: 1em 0 0.5em -0.7em;
border-bottom: 2px dotted blue;
background: #e0e0ff;
}
p {
margin-top: 0;
}
div.linkbar {
width: auto;
text-align: center;
}
div.pagetitle {
width: auto;
text-align: center;
font-weight: bold; font-size: 17pt;
margin-bottom: 10px;
margin-top: 10px;
}
pre.example {
margin-left: 40px;
background: #e0e0e0;
}
div.exampletitle {
font-family: sans-serif;
font-weight: bold;
background: #d0d0f0;
margin-bottom: 0;
}
ul.spaced li { margin-top: 0.7em; }
a.barlink:active { background: yellow; }
div.footer {
margin: 30px -30px -30px -30px;
padding: 5px;
font-size: smaller;
font-style: italic;
}
tt.mem { color: #000030; }

30
wcmtools/memcached/website/memcached.look Executable file
View File

@@ -0,0 +1,30 @@
page<=
{Fps}<html>
<head>
<title>memcached: %%wintitle%%</title>
<link rel='stylesheet' type='text/css' href='memcached.css' />
</head>
<body>
<h1>memcached</h1>
<div class='linkbar'>
<?link ./|About link?>
<?link news.bml|News link?>
<?link download.bml|Download link?>
<?link apis.bml|Client APIs link?>
<?link users.bml|Users link?>
<?link http://lists.danga.com/mailman/listinfo/memcached|Mailing List link?>
</div>
%%body%%
<div class='footer'>Brad Fitzpatrick &lt;brad@danga.com&gt;</div>
</body>
</html>
<=page
h1=>{D}<h2>%%data%%</h2>
memd=>{S}<tt class='mem'>memcached</tt>
link=>{RP}[<a href="%%data1%%" class='barlink'>%%data2%%</a>]

44
wcmtools/memcached/website/news.bml Executable file
View File

@@ -0,0 +1,44 @@
<?page
wintitle=>news
body<=
<?h1 News h1?>
<p><b>2004-04-30:</b> Version 1.1.11 <a
href="download.bml">released</a>. New option (off by default) to not
act as a cache, but hold items forever until deleted (or until machine
goes down, rather). This release also installs the manual page, and
raises the per-process file descriptor limit to match the specified
max connections.
<p><b>2003-12-30:</b> Version 1.1.10 <a
href="download.bml">released</a>. Now includes a command to wipe
the entire cache.
<p><b>2003-10-09:</b> Version 1.1.9 <a
href="download.bml">released</a>, now with better network performance,
better portability, setuid support, and a fix for a rare crash
(SIGPIPE) if a client dies.
<p><b>2003-08-08:</b> We now have <a href="apis.bml">Python support</a>.</p>
<p><b>2003-07-29:</b> Version 1.1.8 <a href="download.bml">released</a>.
The client APIs are now distributed separately.</p>
<p><b>2003-06-30:</b> Version 1.1.6 <a
href="download.bml">released</a>. Many great changes.</p>
<p><b>2003-06-16:</b> <a href="apis.bml">PHP support</a> now included.</p>
<p><b>2003-06-15:</b> Autoconf support added in <a
href="download.bml">1.0.2 release</a>. Website text updated a bit.</p>
<p><b>2003-06-10:</b> Website not so bad now. Files licensed. Perl
API separated and documented. <a href="download.bml">1.0 release</a>.</p>
<p><b>2003-05-30:</b> Server's stable. Perl API's still LJ-specific
(just need to just change the package name, basically). Other APIs
should be easy to bang out. This website needs work.</p>
<=body
page?>

52
wcmtools/memcached/website/users.bml Executable file
View File

@@ -0,0 +1,52 @@
<?page
wintitle=>users
body<=
<?h1 Who's using <?memd?>? h1?>
<p>This is an initial list of <?memd?> users that I've heard about. Please mail me if you're using it, optionally with a little description of how, and I'll add you to this page.</p>
<ul>
<li><b><a href="http://www.livejournal.com/">LiveJournal</a></b> -- fully dynamic blogging site with insane number of unnecessary features, doing over 20 million hits per day. We made <?memd?> for LiveJournal and we hardly ever hit the databases anymore. A few APIs in our codebase still unconditionally hit our databases, but they're being rewritten to be <?memd?>-aware. <?memd?> made a night-and-day difference in the speed of our site.</li>
<li><b><a href="http://www.slashdot.org/">Slashdot</a></b> -- I showed Jamie McCarthy <?memd?> at OSCON 2003 and how we use it on LiveJournal (including our <?memd?>-farm stats page) and he started frothing at the mouth and began implementing it that night in his hotel room. Now Slashdot uses it for caching comments in their rendered form, saving both DB load and web CPU load. They're reportedly working on using <?memd?> in more parts of their code.</li>
<li><b><a href="http://www.wikipedia.org/">WikiPedia</a></b> -- Brion Vibber added support to WikiPedia's MediaWiki backend. ( <a href="http://mail.wikipedia.org/pipermail/wikitech-l/2003-August/005514.html">original announcement</a>).</li>
<li><b><a href="http://vampirefreaks.com">VampireFreaks</a></b>:
<i>"Hey man. I just wanted to thank you for memcached, I just started
using it on http://vampirefreaks.com , a site which gets over a
million page hits a day and has been really slowing down the server.
I've already implemented memcached in a few key spots which determine
the number of users online as well as the number of current users, and
it seems to have helped a lot, I am sure I will be putting it into
more parts of the code as well. Feel free to put us on the memcached
users page if you like."</i></li>
<li><b><a href="http://sourceforge.net">SourceForge</a></b></li>
<li><b><a href="http://www.revelex.com/">Revelex</a></b>: <i>"... We have tried using MySQL, NFS-mounted flat-files and even NFS-mounted RAM drives to no avail. To date, only memcached has been able to keep up with our needs. ..."</i></li>
<li><b><a href="http://www.howardstern.com/">HowardStern.com</a></b>: <i>"We've been struggling to keep the hardware matched to the traffic
level and ever-growing database size. I've now implemented memcached
across major sections of the site and the vBulletin-based forum. We're
using three memcached servers to support the one large mySQL server.
The performance improvement has been tremendous and it allows me to
have an elegant memory caching solution always available instead of
my own cache on the webservers' filesystems, as I had been doing."</i></li>
</ul>
<?h1 Coming soon... h1?>
<p>These people are (or reportedly are, or were) working on memcache support to speed up their sites.</p>
<ul>
<li><b><a href="http://www.everything2.com/">Everything2</a></b> -- <a href="http://lists.danga.com/pipermail/memcached/2003-August/000044.html">adding support</a> for <?memd?> to the ecore nodecache over at <a href="http://www.everydevel.com">everydevel.com</a>.</li>
<li><b><a href="http://www.sourceforge.net/">SourceForge</a></b> -- adding support, which is why the Python API for <?memd?> was created.</li>
</ul>
<=body
page?>