1317 lines
40 KiB
Perl
Executable File
1317 lines
40 KiB
Perl
Executable File
#!/usr/bin/perl
|
|
############################################################################
|
|
|
|
=head1 NAME
|
|
|
|
mogtool -- Inject/extract data to/from a MogileFS installation
|
|
|
|
=head1 SYNOPSIS
|
|
|
|
$ mogtool [general-opts] <command> [command-opts] <command-args>
|
|
|
|
$ mogtool --trackers=127.0.0.1:6001 --domain=foo --class=bar ...
|
|
$ mogtool --conf=foo.conf ...
|
|
|
|
$ mogtool inject thefile.tgz thefilekey
|
|
$ mogtool inject --bigfile thebigfile.tgz thefilekey
|
|
$ mogtool inject --bigfile --gzip thebigfile.tar thefilekey
|
|
$ mogtool inject --bigfile --gzip mydirectory thedirkey
|
|
$ mogtool inject --bigfile --gzip /dev/hda4 thedevkey
|
|
|
|
$ mogtool inject --bigfile --gzip --verify \
|
|
--description="Description" \
|
|
--receipt="foo@bar.com, baz@bar.com" \
|
|
--concurrent=5 --chunksize=32M \
|
|
somehugefile thefilekey
|
|
|
|
$ mogtool extract thefilekey thenewfile.tgz
|
|
$ mogtool extract thefilekey -
|
|
$ mogtool extract --bigfile thedirkey .
|
|
$ mogtool extract --bigfile --asfile thedirkey thefile.tgz
|
|
$ mogtool extract --bigfile thedevkey /dev/hda4
|
|
|
|
$ mogtool delete thekey
|
|
|
|
$ mogtool list
|
|
|
|
=head1 GENERAL OPTIONS
|
|
|
|
=over 4
|
|
|
|
=item --debug
|
|
|
|
Turn on MogileFS debug output.
|
|
|
|
=item --trackers=<[preferred_ip/]ip:port>[,<[preferred_ip/]ip:port>]*
|
|
|
|
Specify one or more trackers for your MogileFS installation. Note that
|
|
you can specify preferred IPs to override the default IPs with. So it
|
|
would look something like B<10.10.0.1/10.0.0.1:8081>.
|
|
|
|
=item --domain=<domain>
|
|
|
|
Set the MogileFS domain to use.
|
|
|
|
=item --class=<class>
|
|
|
|
Set the class within the domain to use. Defaults to _default.
|
|
|
|
=item --conf=<file>
|
|
|
|
Specify a configuration file to load from.
|
|
|
|
=item --lib=<directory>
|
|
|
|
Specify a directory to use as a library path. Right now, this should
|
|
be the directory where you expect to find the MogileFS.pm file, if it's
|
|
not actually installed.
|
|
|
|
=back
|
|
|
|
=head1 COMMANDS
|
|
|
|
=over 4
|
|
|
|
=item inject|i
|
|
|
|
Insert a resource into MogileFS. See L</"INJECT OPTIONS"> and L</"INJECT ARGUMENTS">
|
|
for the rest of how to use the inject mode.
|
|
|
|
=item extract|x
|
|
|
|
Extract a resource from MogileFS. See L</"EXTRACT OPTIONS"> and L</"EXTRACT ARGUMENTS">
|
|
for how to use extract.
|
|
|
|
=item delete|rm
|
|
|
|
Delete a resource. See L</"DELETE OPTIONS"> and L</"DELETE ARGUMENTS">.
|
|
|
|
=item list|ls
|
|
|
|
List all big files contained in MogileFS. No options, no arguments.
|
|
|
|
=back
|
|
|
|
=head1 INJECT OPTIONS
|
|
|
|
The following options are used to control the behavior of the injector.
|
|
|
|
=over 4
|
|
|
|
=item --bigfile|-b
|
|
|
|
If specified, use chunking to break the resource into manageable pieces.
|
|
|
|
=item --chunksize=<size>[B|K|M|G]
|
|
|
|
When instructed to break files into chunks, the injector will use the specified
|
|
chunk size as the maximum chunk size. Defaults to 64M. You can specify the
|
|
chunk size manually and specify the units--defaults to bytes.
|
|
|
|
=item --gzip|-z
|
|
|
|
If specified, mogtool will gzip the data as it's going into MogileFS. The resource
|
|
will be marked as compressed.
|
|
|
|
Note that you do not need to specify this if the resource is already gzipped, but
|
|
it doesn't hurt. (We automatically detect that and mark it as compressed.)
|
|
|
|
=item --overwrite
|
|
|
|
If you previously were working on injecting a big file as chunks and the process
|
|
died, normally mogtool refuses to do it again. Specify this option to force the
|
|
overwrite of that file.
|
|
|
|
B<NOTE:> Other than in the above case (partial failure), mogtool will not prompt
|
|
before overwriting an existing file.
|
|
|
|
=item --verify
|
|
|
|
If on, we do a full MD5 verification of every chunk after it is replicated. This
|
|
can take a while on large files!
|
|
|
|
=item --description=<text>
|
|
|
|
Specifies a description for this file. Optional, but assists in reporting and
|
|
listing the large files in MogileFS. (This is also displayed in any receipts
|
|
that are created.)
|
|
|
|
=item --receipt=<email address>[, <email address>]*
|
|
|
|
If specified, emails a copy of the receipt file to the specified comma-separated
|
|
email addresses. Also creates a local filesystem copy of the receipt file.
|
|
|
|
=item --concurrent=<number>
|
|
|
|
Specifies the number of concurrent processes to run for MogileFS insertion. If
|
|
you are noticing mogtool spend most of it's time waiting for children and not
|
|
actually buffering data, you may wish to raise this number. The default is 1
|
|
but we've found 3 or 4 work well.
|
|
|
|
=item
|
|
|
|
=back
|
|
|
|
=head1 INJECT ARGUMENTS
|
|
|
|
=over 4
|
|
|
|
=item resource
|
|
|
|
What you actually want to inject. This can be a file, directory, or a raw
|
|
partition in the format I</dev/X>.
|
|
|
|
Please see L</"USAGE EXAMPLES"> for more information on how to inject these
|
|
different types of resources and the differences thereof.
|
|
|
|
=item key
|
|
|
|
Specifies the key to save this file to. For big files, the key is actually
|
|
"_big_N:key" and "key,#" where N is one of a bunch of things we use and # is
|
|
the chunk number.
|
|
|
|
Generally, you want this to be descriptive so you remember what it is later
|
|
and can identify the file just by looking at the key.
|
|
|
|
=back
|
|
|
|
=head1 EXTRACT OPTIONS
|
|
|
|
=over 4
|
|
|
|
=item --bigfile|-b
|
|
|
|
If specified, indicates that this resource was chunked on injection and should be
|
|
reassembled for extraction.
|
|
|
|
=item --gzip|-z
|
|
|
|
Specifies to mogtool that it should ungzip the output if and only if it was
|
|
compressed when inserted into the MogileFS system. So, if you're extracting a
|
|
file that wasn't gzipped to begin with, this doesn't do anything.
|
|
|
|
=item --asfile
|
|
|
|
Useful when extracting something previously inserted as a directory--this option
|
|
instructs mogtool to treat the resource as a file and not actually run it
|
|
through tar for decompression.
|
|
|
|
=back
|
|
|
|
=head1 EXTRACT ARGUMENTS
|
|
|
|
=over 4
|
|
|
|
=item key
|
|
|
|
Specifies the key to get the file from.
|
|
|
|
=item destination
|
|
|
|
What destination means varies depending on what type of resource you're extracting.
|
|
However, no matter what, you can specify a single dash (B<->) to mean STDOUT.
|
|
|
|
Please see the usage examples for more information on how extract works.
|
|
|
|
=back
|
|
|
|
=head1 DELETE OPTIONS
|
|
|
|
=over 4
|
|
|
|
=item --bigfile|-b
|
|
|
|
The resource is a "big file" and all chunks should be deleted.
|
|
|
|
=back
|
|
|
|
=head1 DELETE ARGUMENTS
|
|
|
|
=over 4
|
|
|
|
=item key
|
|
|
|
Specifies the key of the file to delete.
|
|
|
|
=back
|
|
|
|
=head1 USAGE EXAMPLES
|
|
|
|
I<Please note that all examples assume you have a default config file that
|
|
contains the tracker and domain to use. Saves us from having to clutter up
|
|
the command line.>
|
|
|
|
=head2 Small Files (<64MB)
|
|
|
|
When it comes to using small files, mogtool is very, very easy.
|
|
|
|
=head3 Injection
|
|
|
|
$ mogtool inject foo.dbm foo.dbm.2004.12
|
|
|
|
Injects the file I<foo.dbm> into MogileFS under the key of I<foo.dbm.2004.12>.
|
|
|
|
$ mogtool inject --gzip foo.dbm foo.dbm.2004.12
|
|
|
|
Injects the same file to the same key, but compresses it on the fly for you.
|
|
|
|
=head3 Extraction
|
|
|
|
$ mogtool extract foo.dbm.2004.12 newfoo.dbm
|
|
|
|
Retrieves the key I<foo.dbm.2004.12> and saves it as I<newfoo.dbm>.
|
|
|
|
$ mogtool extract --gzip foo.dbm.2004.12 newfoo.dbm
|
|
|
|
Gets the file and automatically decompresses it, if and only if it was compressed.
|
|
So basically, you can turn on gzip in your config file and mogtool will do the
|
|
smart thing each time.
|
|
|
|
$ mogtool extract foo.dbm.2004.12 -
|
|
|
|
Print the resource to standard out. If you want, you can pipe it somewhere or
|
|
redirect to a file (but why not just specify the filename?).
|
|
|
|
=head2 Large Files (>64MB)
|
|
|
|
Given mogtool's ability to break files into chunks and later reassemble them,
|
|
inserting large files (even files over the 4GB barrier) is relatively easy.
|
|
|
|
=head3 Injection
|
|
|
|
$ mogtool inject --bigfile largefile.dat largefile.dat
|
|
|
|
As expected, inserts the file I<largefile.dat> into the MogileFS system under
|
|
the name I<largefile.dat>. Not very creative. Uses the default 64MB chunks.
|
|
|
|
$ mogtool inject --bigfile --chunksize=16M largefile.dat largefile.dat
|
|
|
|
Specify to use 16MB chunks instead of the default. Otherwise, the same.
|
|
|
|
$ mogtool inject --bigfile --chunksize=1000K --gzip largefile.dat somekey
|
|
|
|
Do it again, but specify 1000KB chunks, gzip automatically, and upload it under
|
|
a different key I<somekey>.
|
|
|
|
$ mogtool inject --bigfile --concurrent=5 --gzip largefile.dat somekey
|
|
|
|
Same as above, but use 5 children processes for uploading chunks to MogileFS.
|
|
This can take up to 300MB of memory in this example! (It tends to use about
|
|
(concurrency + 1) * chunksize bytes.)
|
|
|
|
$ mogtool inject --bigfile --chunksize=32M --concurrent=3 --gzip \
|
|
--receipt="foo@bar.com" --verify --description="A large file" \
|
|
largefile.dat somekey
|
|
|
|
Break this file into 128MB chunks, set a description, use 3 children to
|
|
upload them, gzip the file as you go, do a full MD5 verification of every
|
|
chunk, then email a receipt with all of the MogileFS paths to me.
|
|
|
|
Lots of flexibility with mogtool.
|
|
|
|
=head3 Extraction
|
|
|
|
$ mogtool extract --bigfile somekey newfile.dat
|
|
|
|
In its basic form, extracts the previously inserted large file and saves it as
|
|
I<newfile.dat>.
|
|
|
|
$ mogtool extract --bigfile --gzip somekey newfile.dat
|
|
|
|
If the file was gzipped on entry, ungzip it and save the result. If it wasn't
|
|
gzipped, then we just save it.
|
|
|
|
=head2 Directories
|
|
|
|
Directories are easily injected and extracted with mogtool. To create the data
|
|
stream that is inserted into MogileFS, we use tar.
|
|
|
|
=head3 Injection
|
|
|
|
$ mogtool inject --bigfile mydir mykey
|
|
|
|
Run I<mydir> through tar and then save it as I<mykey>.
|
|
|
|
$ mogtool inject --bigfile --gzip --concurrent=5 mydir mykey
|
|
|
|
Inject, but also gzip and use multiple injectors.
|
|
|
|
I<Note how this is just like injecting a large file. See injection examples for
|
|
large files for more examples.>
|
|
|
|
=head3 Extraction
|
|
|
|
$ mogtool extract --bigfile mykey .
|
|
|
|
Extract the previously injected directory I<mykey> to your local directory.
|
|
|
|
$ mogtool extract --bigfile --asfile mykey foo.tar
|
|
|
|
Take the previously generated tarball and save it as I<foo.tar>. Simply creates
|
|
the file instead of extracting everything inside.
|
|
|
|
=head2 Partitions/Devices
|
|
|
|
mogtool has the ability to inject raw partitions into MogileFS and to retrieve
|
|
them later and write them back to a partition. They're treated just like directories
|
|
for the most part, we just don't pipe things through tar.
|
|
|
|
=head3 Injection
|
|
|
|
$ mogtool inject --bigfile /dev/hda3 hda3.backup
|
|
|
|
Save a raw copy of your partition I</dev/hda3> to the key I<hda3.backup>.
|
|
|
|
$ mogtool inject --bigfile --gzip /dev/hda3 hda3.backup
|
|
|
|
Same, but compress on the fly during injection.
|
|
|
|
=head3 Extraction
|
|
|
|
$ mogtool extract --bigfile hda3.backup /dev/hda4
|
|
|
|
Extract the partition at I<hda3.backup> to the partition I</dev/hda4>. B<WARNING:>
|
|
mogtool won't ask for confirmation, make sure you don't mistype partition numbers!
|
|
|
|
=head2 Deleting a Resource
|
|
|
|
B<WARNING:> Please make sure you're specifying the right parameter, as delete does
|
|
not prompt for confirmation of the request!
|
|
|
|
$ mogtool delete thekey
|
|
|
|
Delete a normal file.
|
|
|
|
$ mogtool delete --bigfile thekey
|
|
|
|
Delete a chunked file--this deletes all chunks and the receipt, so the file is gone.
|
|
|
|
=head2 Listing Big Files
|
|
|
|
$ mogtool list backup
|
|
|
|
Lists all large files stored in MogileFS. It is not possible to list all normal files
|
|
at this time.
|
|
|
|
=head1 CONFIGURATION FILE
|
|
|
|
Instead of adding a ton of options to the command line every time, mogtool enables
|
|
you to create a default configuration file that it will read all of the options from.
|
|
It searches two locations for a default configuration file: B<~/.mogtool> and
|
|
B</etc/mogilefs/mogtool.conf>. (Alternately, you can specify B<--conf=whatever> as
|
|
an option on the command line.)
|
|
|
|
The file can consist of any number of the following items:
|
|
|
|
trackers = 10.0.0.3:7001, 10.10.0.5/10.0.0.5:7001
|
|
domain = mogiledomain
|
|
class = fileclass
|
|
lib = /home/foo/lib
|
|
gzip = 1
|
|
big = 1
|
|
overwrite = 1
|
|
chunksize = 32M
|
|
receipt = foo@bar.com, baz@bar.com
|
|
verify = 1
|
|
concurrent = 3
|
|
|
|
=head1 KNOWN BUGS
|
|
|
|
None? Send me any you find! :)
|
|
|
|
=head1 PLANNED FEATURES
|
|
|
|
=over 4
|
|
|
|
=item --concurrent for extract
|
|
|
|
It would be nice to have concurrent extraction going on.
|
|
|
|
=item recover mode
|
|
|
|
If the receipt file is ever corrupt in MogileFS it would be useful to recover a
|
|
file given just a receipt. It would have the same arguments as the extract mode,
|
|
except use a receipt file as the data source.
|
|
|
|
=item partition size verification
|
|
|
|
We can easily get the partition size when we save one to MogileFS, so we should
|
|
use that information to determine during extraction if a target partition is going
|
|
to be big enough.
|
|
|
|
=item on the fly gzip extraction
|
|
|
|
Right now we can gzip on an injection, but we should support doing decompression
|
|
on the fly coming out of MogileFS.
|
|
|
|
=item make list take a prefix
|
|
|
|
If you can specify a prefix, that makes things easier for finding small files that
|
|
are stored in MogileFS.
|
|
|
|
=item more information on list
|
|
|
|
Have list load up the info file and parse it for information about each of the
|
|
big files being stored. Maybe have this as an option (-l). (This means the
|
|
reading and parsing of info files should be abstracted into a function.)
|
|
|
|
=back
|
|
|
|
=head1 AUTHOR
|
|
|
|
Mark Smith E<lt>junior@danga.comE<gt> - most of the implementation and maintenance.
|
|
|
|
Brad Fitzpatrick E<lt>brad@danga.comE<gt> - concepts and rough draft.
|
|
|
|
Copyright (c) 2002-2004 Danga Interactive. All rights reserved.
|
|
|
|
=cut
|
|
|
|
##############################################################################
|
|
|
|
use strict;
|
|
use Getopt::Long;
|
|
use Pod::Usage qw{ pod2usage };
|
|
use Digest::MD5 qw{ md5_hex };
|
|
use Time::HiRes qw{ gettimeofday tv_interval };
|
|
use LWP::Simple;
|
|
use POSIX qw(:sys_wait_h);
|
|
use Compress::Zlib;
|
|
|
|
$| = 1;
|
|
|
|
use constant ERR_FATAL => 1;
|
|
|
|
my %opts;
|
|
$opts{help} = 0;
|
|
|
|
abortWithUsage() unless
|
|
GetOptions(
|
|
# general purpose options
|
|
'trackers=s' => \$opts{trackers},
|
|
'domain=s' => \$opts{domain},
|
|
'class=s' => \$opts{class},
|
|
'config=s' => \$opts{conf},
|
|
'help' => \$opts{help},
|
|
'debug' => \$MogileFS::DEBUG,
|
|
'lib' => \$opts{lib},
|
|
|
|
# extract+inject options
|
|
'gzip|z' => \$opts{gzip},
|
|
'bigfile|b' => \$opts{big},
|
|
|
|
# inject options
|
|
'overwrite' => \$opts{overwrite},
|
|
'chunksize=s' => \$opts{chunksize},
|
|
'receipt=s' => \$opts{receipt},
|
|
'reciept=s' => \$opts{receipt}, # requested :)
|
|
'verify' => \$opts{verify},
|
|
'description=s' => \$opts{des},
|
|
'concurrent=i' => \$opts{concurrent},
|
|
|
|
# extract options
|
|
'asfile' => \$opts{asfile},
|
|
);
|
|
|
|
# now load the config file?
|
|
my @confs = ( $opts{conf}, "$ENV{HOME}/.mogtool", "/etc/mogilefs/mogtool.conf" );
|
|
foreach my $conf (@confs) {
|
|
next unless $conf && -e $conf;
|
|
open FILE, "<$conf";
|
|
foreach (<FILE>) {
|
|
s!#.*!!;
|
|
next unless m!(\w+)\s*=\s*(.+)!;
|
|
$opts{$1} = $2;
|
|
}
|
|
close FILE;
|
|
}
|
|
|
|
# now bring in MogileFS, because hopefully we have a lib by now
|
|
if ($opts{lib}) {
|
|
eval "use lib '$opts{lib}';";
|
|
}
|
|
eval "use MogileFS;";
|
|
|
|
# no trackers and domain..?
|
|
unless ($opts{trackers} && $opts{domain}) {
|
|
abortWithUsage();
|
|
}
|
|
|
|
# init connection to mogile
|
|
my $mogfs = get_mogfs();
|
|
|
|
# get our command and pass off to our functions
|
|
my $cmd = shift;
|
|
inject() if $cmd eq 'i' || $cmd eq "inject";
|
|
extract() if $cmd eq 'x' || $cmd eq "extract";
|
|
list() if $cmd eq 'ls' || $cmd eq "list";
|
|
mdelete() if $cmd eq 'rm' || $cmd eq "delete";
|
|
|
|
# fail if we get this far
|
|
abortWithUsage();
|
|
|
|
######################################################################
|
|
|
|
sub get_mogfs {
|
|
my @trackerinput = split(/\s*,\s*/, $opts{trackers});
|
|
my @trackers;
|
|
my %pref_ip;
|
|
foreach my $tracker (@trackerinput) {
|
|
if ($tracker =~ m!(.+)/(.+):(\d+)!) {
|
|
$pref_ip{$2} = $1;
|
|
push @trackers, "$2:$3";
|
|
} else {
|
|
push @trackers, $tracker;
|
|
}
|
|
}
|
|
|
|
my $mogfs = MogileFS->new(
|
|
domain => $opts{domain},
|
|
hosts => \@trackers,
|
|
)
|
|
or error("Could not initialize MogileFS", ERR_FATAL);
|
|
$mogfs->set_pref_ip(\%pref_ip);
|
|
return $mogfs;
|
|
}
|
|
|
|
sub error {
|
|
my $err = shift() || "ERROR: no error message provided!";
|
|
print STDERR "$err\n";
|
|
|
|
if (my $errstr = $mogfs->errstr) {
|
|
$errstr =~ s/^\s+//;
|
|
$errstr =~ s/\s+$//;
|
|
if ($errstr) {
|
|
print STDERR "MogileFS backend error message: $errstr\n";
|
|
}
|
|
}
|
|
|
|
if ($@) {
|
|
my $err = $@;
|
|
$err =~ s/[\r\n]+$//;
|
|
print STDERR "System error message: $@\n";
|
|
}
|
|
|
|
# if a second argument, exit
|
|
if (defined (my $exitcode = shift())) {
|
|
exit $exitcode+0;
|
|
}
|
|
}
|
|
|
|
sub inject {
|
|
my $src = shift @ARGV;
|
|
my $key = shift @ARGV;
|
|
abortWithUsage() unless $src && $key;
|
|
|
|
# make sure the source exists and the key is valid
|
|
die "Error: source $src doesn't exist.\n"
|
|
unless -e $src;
|
|
die "Error: key $key isn't valid; must not contain spaces or commas.\n"
|
|
unless $key =~ /^[^\s\,]+$/;
|
|
|
|
# before we get too far, find sendmail?
|
|
my $sendmail;
|
|
if ($opts{receipt}) {
|
|
$sendmail = `which sendmail` || '/usr/sbin/sendmail';
|
|
$sendmail =~ s/[\r\n]+$//;
|
|
unless (-e $sendmail) {
|
|
die "Error: attempted to find sendmail binary in /usr/sbin but couldn't.\n";
|
|
}
|
|
}
|
|
|
|
# open up O as the handle to use for reading data
|
|
my $type = 'unknown';
|
|
if (-d $src) {
|
|
my $taropts = ($opts{gzip} ? 'z' : '') . "cf";
|
|
$type = 'tarball';
|
|
open (O, '-|', 'tar', $taropts, '-', $src)
|
|
or die "Couldn't open tar for reading: $!\n";
|
|
} elsif (-f $src) {
|
|
$type = 'file';
|
|
open (O, "<$src")
|
|
or die "Couldn't open file for reading: $!\n";
|
|
} elsif (-b $src) {
|
|
$type = 'partition';
|
|
open (O, "<$src")
|
|
or die "Couldn't open block device for reading: $!\n";
|
|
} else {
|
|
die "Error: not file, directory, or partition.\n";
|
|
}
|
|
|
|
# now do some pre-file checking...
|
|
my $size = -s $src;
|
|
if ($type ne 'file') {
|
|
die "Error: you specified to store a file of type $type but didn't specify --bigfile. Please see documentation.\n"
|
|
unless $opts{big};
|
|
} elsif ($size > 64 * 1024 * 1024) {
|
|
die "Error: the file is more than 64MB and you didn't specify --bigfile. Please see documentation.\n"
|
|
unless $opts{big};
|
|
}
|
|
|
|
# see if there's already a pre file?
|
|
if ($opts{big}) {
|
|
my $data = $mogfs->get_file_data("_big_pre:$key");
|
|
if (defined $data) {
|
|
unless ($opts{overwrite}) {
|
|
error(<<MSG, ERR_FATAL);
|
|
ERROR: The pre-insert file for $key exists. This indicates that a previous
|
|
attempt to inject a file failed--or is still running elsewhere! Please
|
|
verify that a previous injection of this file is finished, or run mogtool
|
|
again with the --overwrite inject option.
|
|
|
|
$$data
|
|
MSG
|
|
}
|
|
|
|
# delete the pre notice since we didn't die (overwrite must be on)
|
|
$mogfs->delete("_big_pre:$key")
|
|
or error("ERROR: Unable to delete _big_pre:$key.", ERR_FATAL);
|
|
}
|
|
|
|
# now create our pre notice
|
|
my $prefh = $mogfs->new_file("_big_pre:$key", $opts{class})
|
|
or error("ERROR: Unable to create _big_pre:$key.", ERR_FATAL);
|
|
$prefh->print("starttime:" . time());
|
|
$prefh->close()
|
|
or error("ERROR: Unable to save to _big_pre:$key.", ERR_FATAL);
|
|
}
|
|
|
|
# setup config and temporary variables we're going to be using
|
|
my $chunk_size = 64 * 1024 * 1024; # 64 MB
|
|
if ($opts{big}) {
|
|
if ($opts{chunksize} && ($opts{chunksize} =~ m!^(\d+)(G|M|K|B)?!i)) {
|
|
$chunk_size = $1;
|
|
unless (lc $2 eq 'b') {
|
|
$chunk_size *= (1024 ** ( { g => 3, m => 2, k => 1 }->{lc $2} || 2 ));
|
|
}
|
|
print "NOTE: Using chunksize of $chunk_size bytes.\n";
|
|
}
|
|
}
|
|
my $read_size = ($chunk_size > 1024*1024 ? 1024*1024 : $chunk_size);
|
|
|
|
# temporary variables
|
|
my $buf;
|
|
my $bufsize = 0;
|
|
my $chunknum = 0;
|
|
my %chunkinfo; # { id => [ md5, length ] }
|
|
my %chunkbuf; # { id => data }
|
|
my %children; # { pid => chunknum }
|
|
my %chunksout; # { chunknum => pid }
|
|
|
|
# this function writes out a chunk
|
|
my $emit = sub {
|
|
my $cn = shift() + 0;
|
|
return unless $cn;
|
|
|
|
# get the length of the chunk we're going to send
|
|
my $bufsize = length $chunkbuf{$cn};
|
|
return unless $bufsize;
|
|
|
|
# now spawn off a child to do the real work
|
|
if (my $pid = fork()) {
|
|
print "Spawned child $pid to deal with chunk number $cn.\n";
|
|
$chunksout{$cn} = $pid;
|
|
$children{$pid} = $cn;
|
|
return;
|
|
}
|
|
|
|
# drop other memory references we're not using anymore
|
|
foreach my $chunknum (keys %chunkbuf) {
|
|
next if $chunknum == $cn;
|
|
delete $chunkbuf{$chunknum};
|
|
}
|
|
|
|
# as a child, get a new mogile connection
|
|
my $mogfs = get_mogfs();
|
|
my $dkey = $opts{big} ? "$key,$chunknum" : "$key";
|
|
|
|
# TODO: be resilient to transient errors, retry, etc.
|
|
my $start_time = [ gettimeofday() ];
|
|
my $try = 0;
|
|
while (1) {
|
|
$try++;
|
|
my $fh = $mogfs->new_file($dkey, $opts{class}, $bufsize);
|
|
unless (defined $fh) {
|
|
error("WARNING: Unable to create new file '$dkey'.");
|
|
printf "This was try #$try and it's been %.2f seconds since we first tried. Retrying...\n", tv_interval($start_time);
|
|
sleep 1;
|
|
next;
|
|
}
|
|
$fh->print($chunkbuf{$cn});
|
|
unless ($fh->close) {
|
|
error("WARNING: Unable to save file '$dkey'.");
|
|
printf "This was try #$try and it's been %.2f seconds since we first tried. Retrying...\n", tv_interval($start_time);
|
|
sleep 1;
|
|
next;
|
|
}
|
|
last;
|
|
}
|
|
my $diff = tv_interval($start_time);
|
|
printf " chunk $cn saved in %.2f seconds.\n", $diff;
|
|
|
|
# make sure we never return, always exit
|
|
exit 0;
|
|
};
|
|
|
|
# just used to reap our children in a loop until they're done. also
|
|
# handles respawning a child that failed.
|
|
my $reap_children = sub {
|
|
# find out if we have any kids dead
|
|
while ((my $pid = waitpid -1, WNOHANG) > 0) {
|
|
my $cnum = delete $children{$pid};
|
|
unless ($cnum) {
|
|
print "Error: reaped child $pid, but no idea what they were doing...\n";
|
|
next;
|
|
}
|
|
if (my $status = $?) {
|
|
print "Error: reaped child $pid for chunk $cnum returned non-zero status... Retrying...\n";
|
|
$emit->($cnum);
|
|
next;
|
|
}
|
|
my @paths = grep { defined $_ } $mogfs->get_paths($opts{big} ? "$key,$cnum" : "$key", 1);
|
|
unless (@paths) {
|
|
print "Error: reaped child $pid for chunk $cnum but no paths exist... Retrying...\n";
|
|
$emit->($cnum);
|
|
next;
|
|
}
|
|
delete $chunkbuf{$cnum};
|
|
delete $chunksout{$cnum};
|
|
print "Child $pid successfully finished with chunk $cnum.\n";
|
|
}
|
|
};
|
|
|
|
# this function handles parallel threads
|
|
$opts{concurrent} ||= 1;
|
|
$opts{concurrent} = 1 if $opts{concurrent} < 1;
|
|
my $handle_children = sub {
|
|
# here we pause while our children are working
|
|
my $first = 1;
|
|
while ($first || scalar(keys %children) >= $opts{concurrent}) {
|
|
$first = 0;
|
|
$reap_children->();
|
|
select undef, undef, undef, 0.1;
|
|
}
|
|
|
|
# now spawn until we hit the limit
|
|
foreach my $cnum (keys %chunkbuf) {
|
|
next if $chunksout{$cnum};
|
|
$emit->($cnum);
|
|
last if scalar(keys %children) >= $opts{concurrent};
|
|
}
|
|
};
|
|
|
|
# setup compression stuff
|
|
my $dogzip = 0;
|
|
my $zlib;
|
|
if ($opts{gzip}) {
|
|
# if they turned gzip on we may or may not need this stream, so make it
|
|
$zlib = deflateInit()
|
|
or error("Error: unable to create gzip deflation stream", ERR_FATAL);
|
|
}
|
|
|
|
# read one meg chunks while we have data
|
|
my $sum = 0;
|
|
my $readbuf = '';
|
|
while (my $rv = read(O, $readbuf, $read_size)) {
|
|
# if this is a file, and this is our first read, see if it's gzipped
|
|
if (!$sum && $rv >= 2) {
|
|
if (substr($readbuf, 0, 2) eq "\x1f\x8b") {
|
|
# this is already gzipped, so just mark it as such and insert it
|
|
$opts{gzip} = 1;
|
|
} else {
|
|
# now turn on our gzipping if the user wants the output gzipped
|
|
$dogzip = 1 if $opts{gzip};
|
|
}
|
|
}
|
|
|
|
# now run it through the deflation stream before we process it here
|
|
if ($dogzip) {
|
|
my ($out, $status) = $zlib->deflate($readbuf);
|
|
error("Error: Deflation failure processing stream", ERR_FATAL)
|
|
unless $status == Z_OK;
|
|
$readbuf = $out;
|
|
$rv = length $readbuf;
|
|
|
|
# we don't always get a chunk from deflate
|
|
next unless $rv;
|
|
}
|
|
|
|
# now stick our data into our real buffer
|
|
$buf .= $readbuf;
|
|
$bufsize += $rv;
|
|
$sum += $rv;
|
|
$readbuf = '';
|
|
|
|
# generate output
|
|
if ($type ne 'tarball' && $size && $size > $read_size) {
|
|
printf "Buffer so far: $bufsize bytes [%.2f%% complete]\r", ($sum / $size * 100);
|
|
} else {
|
|
print "Buffer so far: $bufsize bytes\r";
|
|
}
|
|
|
|
# if we have one chunk, handle it
|
|
if ($bufsize >= $chunk_size) {
|
|
$chunkbuf{++$chunknum} = substr($buf, 0, $chunk_size);
|
|
|
|
# calculate the md5, print out status, and save this chunk
|
|
my $md5 = md5_hex($buf);
|
|
if ($opts{big}) {
|
|
print "chunk $key,$chunknum: $md5, len = $chunk_size\n";
|
|
} else {
|
|
print "file $key: $md5, len = $chunk_size\n";
|
|
}
|
|
$chunkinfo{$chunknum} = [ $md5, $chunk_size ];
|
|
|
|
# reset for the next read loop
|
|
$buf = substr($buf, $chunk_size);
|
|
$bufsize = length $buf;
|
|
|
|
# now spawn children to save chunks
|
|
$handle_children->();
|
|
}
|
|
}
|
|
close O;
|
|
|
|
# now we need to flush the gzip engine
|
|
if ($dogzip) {
|
|
my ($out, $status) = $zlib->flush;
|
|
error("Error: Deflation failure processing stream", ERR_FATAL)
|
|
unless $status == Z_OK;
|
|
$buf .= $out;
|
|
$bufsize += length $out;
|
|
$sum += length $out;
|
|
}
|
|
|
|
# final piece
|
|
if ($buf) {
|
|
$chunkbuf{++$chunknum} = $buf;
|
|
my $md5 = md5_hex($buf);
|
|
if ($opts{big}) {
|
|
print "chunk $key,$chunknum: $md5, len = $bufsize\n";
|
|
} else {
|
|
print "file $key: $md5, len = $bufsize\n";
|
|
}
|
|
$chunkinfo{$chunknum} = [ $md5, $bufsize ];
|
|
}
|
|
|
|
# now, while we still have chunks to process...
|
|
while (%chunkbuf) {
|
|
$handle_children->();
|
|
sleep 1;
|
|
}
|
|
|
|
# verify replication and chunks
|
|
my %paths; # { chunknum => [ path, path, path ... ] }
|
|
my %still_need = ( %chunkinfo );
|
|
while (%still_need) {
|
|
print "Beginning replication wait: " . join(' ', sort { $a <=> $b } keys %still_need) . "\n";
|
|
sleep 1; # give things time to replicate some
|
|
|
|
# now iterate over each and get the paths
|
|
foreach my $num (keys %still_need) {
|
|
my $dkey = $opts{big} ? "$key,$num" : $key;
|
|
my @npaths = grep { defined $_ } $mogfs->get_paths($dkey, 1);
|
|
|
|
unless (@npaths) {
|
|
error("FAILURE: chunk $num has no paths at all.", ERR_FATAL);
|
|
}
|
|
|
|
if (scalar(@npaths) >= 2) {
|
|
# okay, this one's replicated, actually verify the paths
|
|
foreach my $path (@npaths) {
|
|
if ($opts{verify}) {
|
|
print " Verifying chunk $num, path $path...";
|
|
my $data = get($path);
|
|
my $len = length($data);
|
|
my $md5 = md5_hex($data);
|
|
if ($md5 ne $chunkinfo{$num}->[0]) {
|
|
print "md5 mismatch\n";
|
|
next;
|
|
} elsif ($len != $chunkinfo{$num}->[1]) {
|
|
print "length mismatch ($len, $chunkinfo{$num}->[1])\n";
|
|
next;
|
|
}
|
|
print "ok\n";
|
|
} elsif ($opts{receipt}) {
|
|
# just do a quick size check
|
|
print " Size verifying chunk $num, path $path...";
|
|
my $clen = (head($path))[1] || 0;
|
|
unless ($clen == $chunkinfo{$num}->[1]) {
|
|
print "length mismatch ($clen, $chunkinfo{$num}->[1])\n";
|
|
next;
|
|
}
|
|
print "ok\n";
|
|
}
|
|
push @{$paths{$num} ||= []}, $path;
|
|
}
|
|
|
|
# now make sure %paths contains at least 2 verified
|
|
next unless scalar(@{$paths{$num} || []}) >= 2;
|
|
delete $still_need{$num};
|
|
}
|
|
}
|
|
}
|
|
|
|
# prepare the info file
|
|
my $des = $opts{des} || 'no description';
|
|
my $compressed = $opts{gzip} ? '1' : '0';
|
|
#FIXME: add 'partblocks' to info file
|
|
|
|
# create the info file
|
|
my $info = <<INFO;
|
|
des $des
|
|
type $type
|
|
compressed $compressed
|
|
filename $src
|
|
chunks $chunknum
|
|
size $sum
|
|
|
|
INFO
|
|
foreach (sort { $a <=> $b } keys %chunkinfo) {
|
|
$info .= "part $_ bytes=$chunkinfo{$_}->[1] md5=$chunkinfo{$_}->[0] paths: ";
|
|
$info .= join(', ', @{$paths{$_} || []});
|
|
$info .= "\n";
|
|
}
|
|
|
|
# now write out the info file
|
|
if ($opts{big}) {
|
|
my $fhinfo = $mogfs->new_file("_big_info:$key", $opts{class})
|
|
or error("ERROR: Unable to create _big_info:$key.", ERR_FATAL);
|
|
$fhinfo->print($info);
|
|
$fhinfo->close()
|
|
or error("ERROR: Unable to save _big_info:$key.", ERR_FATAL);
|
|
|
|
# verify info file
|
|
print "Waiting for info file replication...\n";
|
|
while (1) {
|
|
my @paths = $mogfs->get_paths("_big_info:$key", 1);
|
|
next unless scalar(@paths) >= 2;
|
|
foreach my $path (@paths) {
|
|
my $data = get($path);
|
|
error(" FATAL: content mismatch on $path", ERR_FATAL)
|
|
unless $data eq $info;
|
|
}
|
|
last;
|
|
}
|
|
|
|
# now delete our pre file
|
|
print "Deleting pre-insert file...\n";
|
|
$mogfs->delete("_big_pre:$key")
|
|
or error("ERROR: Unable to delete _big_pre:$key", ERR_FATAL);
|
|
}
|
|
|
|
# now email and save a receipt
|
|
if ($opts{receipt}) {
|
|
open MAIL, "| $sendmail -t"
|
|
or error("ERROR: Unable to open sendmail binary: $sendmail", ERR_FATAL);
|
|
print MAIL <<MAIL;
|
|
To: $opts{receipt}
|
|
From: mogtool\@dev.null
|
|
Subject: mogtool.$key.receipt
|
|
|
|
$info
|
|
.
|
|
MAIL
|
|
close MAIL;
|
|
print "Receipt emailed.\n";
|
|
|
|
# now dump to a file
|
|
open FILE, ">mogtool.$key.receipt"
|
|
or error("ERROR: Unable to create file mogtool.$key.receipt in current directory.", ERR_FATAL);
|
|
print FILE $info;
|
|
close FILE;
|
|
print "Receipt stored in mogtool.$key.receipt.\n";
|
|
}
|
|
|
|
exit 0;
|
|
}
|
|
|
|
sub _parse_info {
|
|
my $info = shift;
|
|
my $res = {};
|
|
|
|
# parse out the header data
|
|
$res->{des} = ($info =~ /^des\s+(.+)$/m) ? $1 : undef;
|
|
$res->{type} = ($info =~ /^type\s+(.+)$/m) ? $1 : undef;
|
|
$res->{compressed} = ($info =~ /^compressed\s+(.+)$/m) ? $1 : undef;
|
|
$res->{filename} = ($info =~ /^filename\s+(.+)$/m) ? $1 : undef;
|
|
$res->{chunks} = ($info =~ /^chunks\s+(\d+)$/m) ? $1 : undef;
|
|
$res->{size} = ($info =~ /^size\s+(\d+)$/m) ? $1 : undef;
|
|
|
|
# now get the pieces
|
|
$res->{maxnum} = undef;
|
|
while ($info =~ /^part\s+(\d+)\s+bytes=(\d+)\s+md5=(.+)\s+paths:\s+(.+)$/mg) {
|
|
$res->{maxnum} = $1 if !defined $res->{maxnum} || $1 > $res->{maxnum};
|
|
$res->{parts}->{$1} = {
|
|
bytes => $2,
|
|
md5 => $3,
|
|
paths => [ split(/\s*,\s*/, $4) ],
|
|
};
|
|
}
|
|
|
|
return $res;
|
|
}
|
|
|
|
sub extract {
|
|
my $key = shift @ARGV;
|
|
my $dest = shift @ARGV;
|
|
abortWithUsage() unless $key && $dest;
|
|
|
|
error("Error: key $key isn't valid; must not contain spaces or commas.", ERR_FATAL)
|
|
unless $key =~ /^[^\s\,]+$/;
|
|
unless ($dest eq '-' || $dest eq '.') {
|
|
error("Error: destination exists: $dest (specify --overwrite if you want to kill it)", ERR_FATAL)
|
|
if -e $dest && !$opts{overwrite} && !-b $dest;
|
|
}
|
|
|
|
# see if this is really a big file
|
|
my $file;
|
|
if ($opts{big}) {
|
|
my $info = $mogfs->get_file_data("_big_info:$key");
|
|
die "$key doesn't seem to be a valid big file.\n"
|
|
unless $info && $$info;
|
|
|
|
# verify validity
|
|
$file = _parse_info($$info);
|
|
|
|
# make sure we have enough info
|
|
error("Error: info file doesn't contain the number of chunks", ERR_FATAL)
|
|
unless $file->{chunks};
|
|
error("Error: info file doesn't contain the total size", ERR_FATAL)
|
|
unless $file->{size};
|
|
|
|
} else {
|
|
# not a big file, so it has to be of a certain type
|
|
$file->{type} = 'file';
|
|
$file->{maxnum} = 1;
|
|
$file->{parts}->{1} = {
|
|
paths => [ grep { defined $_ } $mogfs->get_paths($key) ],
|
|
};
|
|
|
|
# now, if it doesn't exist..
|
|
unless (scalar(@{$file->{parts}->{1}->{paths}})) {
|
|
error("Error: file doesn't exist (or did you forget --bigfile?)", ERR_FATAL);
|
|
}
|
|
}
|
|
|
|
# several cases.. going to stdout?
|
|
if ($dest eq '-') {
|
|
*O = *STDOUT;
|
|
} else {
|
|
# open up O as the handle to use for reading data
|
|
if ($file->{type} eq 'file' || $file->{type} eq 'partition' ||
|
|
($file->{type} eq 'tarball' && $opts{asfile})) {
|
|
# just write it to the file with this name, but don't overwrite?
|
|
if ($dest eq '.') {
|
|
$dest = $file->{filename};
|
|
$dest =~ s!^(.+)/!!;
|
|
}
|
|
if (-b $dest) {
|
|
# if we're targetting a block device...
|
|
warn "FIXME: add in block checking\n";
|
|
open O, ">$dest"
|
|
or die "Couldn't open $dest: $!\n";
|
|
} elsif (-e $dest) {
|
|
if ($opts{overwrite}) {
|
|
open O, ">$dest"
|
|
or die "Couldn't open $dest: $!\n";
|
|
} else {
|
|
die "File already exists: $dest ... won't overwrite without --overwrite.\n";
|
|
}
|
|
} else {
|
|
open O, ">$dest"
|
|
or die "Couldn't open $dest: $!\n";
|
|
}
|
|
|
|
} elsif ($file->{type} eq 'tarball') {
|
|
my $taropts = ($file->{compressed} ? 'z' : '') . "xf";
|
|
open O, '|-', 'tar', $taropts, '-'
|
|
or die "Couldn't open tar for writing: $!\n";
|
|
|
|
} else {
|
|
die "Error: unable to handle type '$file->{type}'\n";
|
|
}
|
|
}
|
|
|
|
# start fetching pieces
|
|
foreach my $i (1..$file->{maxnum}) {
|
|
print "Fetching piece $i...\n";
|
|
|
|
foreach my $path (@{$file->{parts}->{$i}->{paths} || []}) {
|
|
print " Trying $path...\n";
|
|
my $data = get($path);
|
|
next unless $data;
|
|
|
|
# now verify MD5, etc
|
|
if ($opts{big}) {
|
|
my $len = length $data;
|
|
my $md5 = md5_hex($data);
|
|
print " ($len bytes, $md5)\n";
|
|
next unless $len == $file->{parts}->{$i}->{bytes} &&
|
|
$md5 eq $file->{parts}->{$i}->{md5};
|
|
}
|
|
|
|
# this chunk verified, write it out
|
|
print O $data;
|
|
last;
|
|
}
|
|
}
|
|
|
|
# at this point the file should be complete!
|
|
close O;
|
|
print "Done.\n";
|
|
|
|
# now make sure we have enough data
|
|
#$ mogtool [opts] extract <key> {<file>,<dir>,<device>}
|
|
#=> - (for stdout) (if compressed, add "z" flag)
|
|
#=> . (to untar) (if compressed, do nothing???, make .tar.gz file -- unless they use -z again?)
|
|
#=> /dev/sda4 (but check /proc/partitions that it's big enough) (if compress, Compress::Zlib to ungzip
|
|
# => foo.jpg (write it to a file)
|
|
|
|
|
|
# now check
|
|
exit 0;
|
|
}
|
|
|
|
sub list {
|
|
# list all big files in mogile
|
|
my ($ct, $after, $list);
|
|
while (($after, $list) = $mogfs->list_keys("_big_info:", $after)) {
|
|
last unless $list && @$list;
|
|
|
|
# now extract the key and dump it
|
|
foreach my $key (@$list) {
|
|
next unless $key =~ /^_big_info:(.+)$/;
|
|
|
|
$key = $1;
|
|
$ct++;
|
|
|
|
print "$key\n";
|
|
}
|
|
}
|
|
print "#$ct files found\n";
|
|
exit 0;
|
|
}
|
|
|
|
sub mdelete {
|
|
die "not implemented yet\n";
|
|
}
|
|
|
|
abortWithUsage() if $opts{help};
|
|
|
|
|
|
sub abortWithUsage {
|
|
my $msg = join '', @_;
|
|
|
|
if ( $msg ) {
|
|
pod2usage( -verbose => 1, -exitval => 1, -message => "$msg" );
|
|
} else {
|
|
pod2usage( -verbose => 1, -exitval => 1 );
|
|
}
|
|
}
|
|
|
|
|
|
__END__
|
|
|
|
Usage: mogtool [opts] <command> [command-opts] [command-args]
|
|
|
|
General options:
|
|
* --trackers=<ip:port>[,<ip:port>]*
|
|
|
|
* --domain=<domain>
|
|
|
|
* --class=<class>
|
|
|
|
* --conf=<file> Location of config file listing trackers, default
|
|
domain, and default class
|
|
|
|
Default: ~/.mogilefs, /etc/mogilefs/mogilefs.conf
|
|
|
|
* --bigfile | -b Tell mogtool to split file into 64MB chunks and
|
|
checksum the chunks,
|
|
|
|
* --gzip | -z Use gzip compression/decompression
|
|
|
|
|
|
Commands:
|
|
|
|
inject | i Inject a file into MogileFS, by key
|
|
extract | x Extract a file from MogileFS, by key
|
|
list | ls List large files in MogileFS
|
|
|
|
'inject' syntax:
|
|
|
|
$ mogtool [opts] inject [i-opts] <file,dir,device> <key>
|
|
|
|
Valid i-opts:
|
|
--overwrite Ignore existing _big_pre: and start anew.
|
|
--chunksize=n Set the size of individual chunk files. n is in the format of
|
|
number[scale] so 10 is 10 megabytes, 10M is also 10 megs, 10G, 10B, 10K...
|
|
case insensitive
|
|
--receipt=email Send a receipt to the specified email address
|
|
--verify Make sure things replicate and then check the MD5s?
|
|
--des=string Set the file description
|
|
|
|
|
|
$ mogtool [opts] extract <key> {<file>,<dir>,<device>}
|
|
=> - (for stdout) (if compressed, add "z" flag)
|
|
=> . (to untar) (if compressed, do nothing???, make .tar.gz file -- unless they use -z again?)
|
|
=> /dev/sda4 (but check /proc/partitions that it's big enough) (if compress, Compress::Zlib to ungzip)
|
|
=> foo.jpg (write it to a file)
|
|
|
|
|
|
--key
|
|
|
|
# mogtool add --key='roast.sdb1.2004-11-07' -z /dev/sda1
|
|
|
|
|
|
|
|
<key> = "cow.2004.11.17"
|
|
|
|
# this is a temporary file that we delete when we're doing recording all chunks
|
|
|
|
_big_pre:<key>
|
|
|
|
starttime=UNIXTIMESTAMP
|
|
|
|
# when done, we write the _info file and delete the _pre.
|
|
|
|
_big_info:<key>
|
|
|
|
des Cow's ljdb backup as of 2004-11-17
|
|
type { partition, file, tarball }
|
|
compressed {0, 1}
|
|
filename ljbinlog.305.gz
|
|
partblocks 234324324324
|
|
|
|
|
|
part 1 <bytes> <md5hex>
|
|
part 2 <bytes> <md5hex>
|
|
part 3 <bytes> <md5hex>
|
|
part 4 <bytes> <md5hex>
|
|
part 5 <bytes> <md5hex>
|
|
|
|
_big:<key>,<n>
|
|
_big:<key>,<n>
|
|
_big:<key>,<n>
|
|
|
|
|
|
Receipt format:
|
|
|
|
BEGIN MOGTOOL RECIEEPT
|
|
type partition
|
|
des Foo
|
|
compressed foo
|
|
|
|
part 1 bytes=23423432 md5=2349823948239423984 paths: http://dev5/2/23/23/.fid, http://dev6/23/423/4/324.fid
|
|
part 1 bytes=23423432 md5=2349823948239423984 paths: http://dev5/2/23/23/.fid, http://dev6/23/423/4/324.fid
|
|
part 1 bytes=23423432 md5=2349823948239423984 paths: http://dev5/2/23/23/.fid, http://dev6/23/423/4/324.fid
|
|
part 1 bytes=23423432 md5=2349823948239423984 paths: http://dev5/2/23/23/.fid, http://dev6/23/423/4/324.fid
|
|
|
|
|
|
END RECIEPT
|
|
|
|
|
|
###
|
|
perl -w bin/mogtool --gzip inject --overwrite --chunksize=24M --des="This is a description" --receipt="marksmith@danga.com" ../music/jesse/Unsorted jesse.music.unsorted
|