Subversion backups

At work I have been given the job of administering our Subversion repository for the project.  I am also the “official” expert and go-to guy for Subversion–that ought to be interesting!  I’m no expert on servers and administration, and I only casually use Subversion. The other engineers have not used source code revision yet it seems.  I suppose I will learn as I go.

The subversion repository server is actually just an unused Windows XP workstation that is also acting as a file server.  Since we only need to allow access to the server over the corporate LAN, I decide to use the SVNSERVE to simplify administration and configuration.  I really don’t want to dig into configuring a web server, and I think SVNSERVE will work for us as the team size is still small (about 10 engineers). Because the server is just a regular workstation, I wanted to create a backup system so that if anything happens due to careless mistakes, I can easily bring the repository back to its latest state.

I decided on using a weekly full backup and daily incremental backup system.  I utilized the task scheduling feature of Windows XP to automatically run my backup system in the wee hours of the morning.  It can be accessed and configured to run tasks although a wizard in the Control Panel.  I created one task that will run on Saturday that will perform a full backup of our Subversion repository, and I created another task that will run incremental backups of the Subversion repository on weekdays.

I turned to my trusted tool for doing anything that I find troublesome (repetitive tasks or system administration): Perl.  I ran into early trouble though when I tested the task scheduler with perl.  I used the command “perl.exe -w C:\Perl\scripts\test.pl” with the scheduler, and the scheduled task would run, but my test script did not output to a file to prove that it had run.  I made sure that my %PATH% variable had the Perl binary directory too.  After scratching my head awhile I decided to create a Windows BAT file to execute the perl script.  The BAT file is really simple and looks like this:

@echo off
C:
cd C:\Perl\scripts
@perl.exe -w full_svn_backup.pl

That BAT file did the trick!  I created two BAT files, one that executes a full backup script and one that executes an incremental backup script.  I registered the BAT files with Windows XP task scheduler and then all that was left was to write the backup glue code in the perl scripts. For full backups, one simply has to execute svnadmin dump to backup the entire repository.  The full backup script looks like this:

######################################
# SVN repository full backup script
######################################

use strict;
use Archive::Zip qw( :ERROR_CODES :CONSTANTS );
use Time::localtime;

# determine the current time
my ($day, $month, $year);
my $tm = localtime;
($day, $month, $year) = ($tm->mday, $tm->mon, $tm->year);
$year = $year + 1900;
$month = $month + 1;

# create variables to represent the paths and filenames to the repository and backups
my $repos_dir   = “C:\\svnrepos”;
my $backup_dst  = “C:\\subversion\\backup”;
my $backup_file = “full-backup.” . $year . $month . $day;

# determine the latest revision in the repositry
my $latest_rev = `svnlook youngest $repos_dir`;
chomp($latest_rev);

# execute the svnadmin dump command
my $cmd = “svnadmin dump –revision 0:$latest_rev $repos_dir > $backup_dst\\$backup_file”;
`$cmd`;

# create a ZIP archive of the repository backup to store on backup media
my $zip = Archive::Zip->new();
my $archive = $zip->addFile( “$backup_dst\\$backup_file” );
$zip->writeToFileNamed(“$backup_dst\\$backup_file” . “.zip”);

# save the latest revision number for use by incremental backup script
open(FDREV, “>revinfo.txt”) or die “$!”;
print FDREV $latest_rev . “\n”;
close(FDREV);

I’m not a perl guru who excels at writing greatl perl code, but I know enough to get the job done.  I then created a similar script for performing incremental backups.  Incremental backups are performed by using the –incremental flag with svnadmin dump.

###########################################
# SVN repository incremental backup script
###########################################

use strict;
use Archive::Zip qw( :ERROR_CODES :CONSTANTS );
use Time::localtime;

# determine the current time
my ($day, $month, $year);
my $tm = localtime;
($day, $month, $year) = ($tm->mday, $tm->mon, $tm->year);
$year = $year + 1900;
$month = $month + 1;

# create variables to represent the paths and filenames to the repository and backups
my $repos_dir   = “C:\\svnrepos”;
my $backup_dst  = “C:\\subversion\\backup”;
my $backup_file = “inc-backup.” . $year . $month . $day;

# save the latest full backup revision number for use by incremental back script
open(FDREV, “<revinfo.txt”) or die “$!”;
my $last_rev = <FDREV>;
chomp($last_rev);
close(FDREV);

# determine the latest revision in the repositry
my $latest_rev = `svnlook youngest $repos_dir`;
chomp($latest_rev);

if($latest_rev eq $last_rev){
exit 0;
}

# execute the svnadmin dump command
my $start = $last_rev + 1;
my $end = $latest_rev;
my $cmd = “svnadmin dump –incremental –revision $start:$end $repos_dir > $backup_dst\\$backup_file”;
`$cmd`;

# create a ZIP archive of the repository backup to store on backup media
my $zip = Archive::Zip->new();
my $archive = $zip->addFile( “$backup_dst\\$backup_file” );
$zip->writeToFileNamed(“$backup_dst\\$backup_file” . “.zip”);

# save the latest backup revision number for use by incremental back script
open(FDREV, “>revinfo.txt”) or die “$!”;
print FDREV $latest_rev . “\n”;
close(FDREV);

These two scripts, coupled with BAT files for launching the scripts and Windows XP task scheduler, allow for a convenient backup solution that is really hands off.

Perl – making my life easier since 1999.

Advertisements

News from my desk

So summer is about gone and here I am wondering where the first half of the year has gone.  Autumn is my favorite season though and I’m looking forward to the next few months.  Work is busy and not particularly too interesting.  I’m developing some scheduling software at the very bottom of the MAC layer to interface with the PHY in a next-generation wireless communications system based on WiMAX technology.  The technology will enter trials in Q2 2009 in Japan and we’ll see if it can deliver the data rates we’re hoping for.  Sadly, I suspect this technology will stay here in Japan.  I doubt that Taiwan, Thailand or China will implement this new mobile network with LTE just around the corner as well. They do have the existing PHS infrastructures though, so I won’t bet on anything just yet.

This is my first time to work on MAC layer software.  Until now I’ve worked mainly around the PHY layer.  The MAC layer is not an interesting subject in itself–just a lot of packet formatting and error checking and redundancy.  To all you software people out there, never let the hardware people talk down to you.  Their hardware systems are useless without software in our modern digital communication era.  Don’t get a big head either though, without their hardware improving data rates, we’d still be writing dialers and dealing with TDD access schemes in a wireless medium.

Change of subject, here is a picture of my workstation setup at home.  Please remember I’m in Tokyo and have little space in my tiny apartment for a nicer work area.  C’est la vie.  I dream of a big oak desk with enough room for a desktop workstation and plenty of room to write on paper!

My home workspace
My home workspace

My primary computer is the MacBook on the right.  I use it for e-mail, web browsing, pictures and normal life things.  I also use it to play around with programming stuff related too C++ or Obj-C.  I have some ideas for OS X software, I just have to make the time for them!

On the left is a Sun Blade 100 system, a 64-bit Ultrasparc IIe system I have been using to get a better grasp on the Solaris OS.  Last year I was kind of burnt-out with regard to writing software.  I was thinking of following a SysAdmin track, so I bought the Blade 100 used and studied Solaris.  While I’m not going to become a SysAdmin and I’m going to stick with software, I’m glad I took the time to study Solaris.  The services administration system, svcs, is a wonderful improvement to the UNIX world and I think I would be grumpy if I had to work on Solaris 9 or other UNIX systems that still require scripting to manage services.  I wouldn’t be grumpy actually, I’d just be telling myself how I missed Solaris 10 service administration.  Solaris Zones and ZFS are also really impressive, but I only have a 20 GB disk in the machine and that doesn’t leave me a lot of room to play around.  Currently, I primarily use the machine to fool around with C++ or Erlang these days.  Though Solaris isn’t such a rather poor choice for desktop use, I would happily put it on servers and let it run my software.

I was first introduced to Solaris when I was in college.  The computer engineering department did everything on Windows because we did a lot of FPGA and embedded systems work and the academic versions of the tools only ran on Windows.  I doubt the grad students running the lab had the ability to administer a Linux network either!  I digress.  Our software engineering courses were offered by the computer science department, and all of our assignments had to run on UNIX for grading.  Some people could not stand UNX, but I became attracted to it just because the UNIX lab was always empty and quiet.  Most people would go to the Windows lab, code and test in Visual C++, and then do final testing and submission via via telnet to the UNIX lab machines.  The Windows lab was noisy due to people cranking up their portable CD players (this was before the iPod, folks!) or talking to each other or running large print jobs.  The UNIX lab was always silent and I was able to really focus on my work.   Since then I guess I have developed a soft spot for Solaris and UNIX in general.  It feels the most natural to run g++ and which up some makefiles.  Maybe this is why I like Windows driver development?  When I entered the professional world, getting used to configuring IDEs was a bit of an annoyance.  But that went away fast and now I don’t really care–either way as long as I can build my system and not fight the tools all of the time.   OK, enough nostalgia!

It seems, however, that Solaris is finally good enough for generic x86 systems, so when I move back to the US or move to Taiwan (just a dream!),  I’ll retire the Blade 100 workstation and buy an AMD or Intel workstation so I can run multiple operating systems.  I know Solaris isn’t the zippiest OS, but it has earned my trust over the years and I think that if I had an AMD system with enough memory I probably wouldn’t notice.  After all, anything is an improvement over the Blade 100!

Boost 1.36 and Sun Studio 12

Trying to get Boost and Sun Studio C++ compilers to place nice on Ultrasparc processors isn’t exactly straight-forward.  I fought with both Sun CC and g++, and after browsing mailing lists of Boost and other C++ software, reading various blogs at Sun, and through brute force effort, I managed to build Boost 1.36 for Sun CC.  I just have to add that it was a lot easier getting Boost to build on Mac OS X with g++!  I’m sure someone will say it I’m and idiot since I couldn’t get it to work ont he first try, but let me just say that it was not obvious.  I did a lot of reading and still did not come away with a definitive answer.  The guys at Sun ought to create a Wiki for both Sun Studio users and Sun engineers to share information on working with Boost.

Before I get into the build process, here is the build environment I used just for reference:

# uname -a
SunOS radium 5.10 Generic_137111-02 sun4u sparc SUNW,Sun-Blade-100

# CC -V
CC: Sun C++ 5.9 SunOS_sparc Patch 124863-06 2008/07/08

Of course it is always a good idead to make sure your Sun Studio patches are up-to-date before you start.  The build process I used is listed below.

1. Build a custom version of Bjam for your machine.  Download it from Boost’s sourceforge portal and follow the install directions.  It builds painlessly and installs itself in /usr/bin by default.

2. After downloading and unarchiving Boost 1.36 to a stating platform–I used /opt–change your working directory to the Boost root directory and run the configuration script.  Run ./configure –help to see the various options.  I used the following to specify the location of bjam, the compiler to use, and finally where to put everything in the end:

./configure –with-bjam=/usr/bin/bjam –with-toolset=sun –prefix=/opt

It is important to specify the compiler if you have both g++ and Sun CC in your path environment variable.  On a default developer’s install of Solaris 10, you have g++ in /usr/sfw/bin and then if you installed Sun Studio 12, a symlink is placed in /usr/bin that points to CC.  Be careful!

3. Configure your user-config.jam file which is used by Bjam when building Boost.  After running the configure script, my user-config.jam looked like this:

# Boost.Build Configuration
# Automatically generated by Boost configure

# Compiler configuration
using sun ;

# Python configuration
using python : 2.4 : /usr ;

I added some flags for the compiler to fix a lot of the error messages I was getting when I would run the make process.  The modified file looks like this:

# Boost.Build Configuration
# Automatically generated by Boost configure

# Compiler configuration
using sun : 5.9 : /opt/SUNWspro/bin/CC : <stdlib>sun-stlport <link-flags>-library=stlport4 <cxxflags>-native64 ;

# Python configuration
using python : 2.4 : /usr ;

I should add that I don’t know if this is necessary based on step four in my process.  I really did not feel like trying to hack around the build process as it is complicated and honestly, I want to work on my application, not on becoming a guru Boost builder.  I made this changes because it is suggested on many of the mailing list troubelshooting messages.

4.  Finally, run Bjam to build your application.  On the Boost homepage it says you can run “make install”, but I found whenever I did this the build process would fail to compile any of the libraries.  I explicity invoked the Bjam tool as follows:

bjam toolset=sun stdlib=sun-stlport instruction-set=v9 address-model=64 install

This command may be redundant based on the parameters in user-config.jam, but I just wanted to be sure.  I turned off the monitor and went to bed because it takes a long time to build software on a Sun Blade 100!  Before leaving for work the next morning I checked a log file that I dumbed all of the build process output to and this time I had no errors.  The libraries were put in /opt/lib and my headers in /opt/include.  All is well that ends well.  Now it is time to put Sun’s compiler and tools to work!

Approaching Erlang

Erlang is the tech news more and more these days as a possible tool for designing software to run on multiple core machines.  I’m very interested in servers and high-end server software so I have been following the news to see if Erlang will take off like other languages.  I must admit that I’m not the type to jump onto new languages at a whim.  I’m more intersted in building systems than I am in fancy frameworks or nifty syntax or what not.  Though I’ve considered devoting some time to messing with Eiffel or Ada or Smalltalk or Python, I don’t really have a reason to devote my free (which I prefer to spend on foreign language study) time to them.  C++ and Perl do everything I have ever needed, and I”m already very familiar with the Standard Template Library, POSIX API, Win32 API, and other standard interfaces.  Ultimately when I try a new language, the reason I give up on it is that it takes a lot of time to master new APIs and frameworks.  No matter how many times I tried to do something with Python, it was faster to simply hack it out in C++ even if the code size was much larger.

Erlang does offer something that C++ cannot offer me though: a unique concurrency model without shared memory.  Writing multi-threaded software in C++ is always an exercise in syntax rules, debugging and humility.  Win32 Threads and PThreads are C APIs that force one to battle with static object behavior in your OOP applications.  You might remark that Python and many other languages have threads, why Erlang?  For me, it is all about doing away with shared memory and simplifying the system design.  What’s more, Joe Armstrong, the author of the language, has provided lots of data on the performance of Erlang.  I am also impressed that Ericsson and Nortel, two big cmpanies in the telecom industry I work in, employ Erlang in network telecom equipment that is used in massive telecom systems.  Telecom software is some of the most robust software operating in the wild!

I picked up Joe Armstrong’s text, the only text on Erlang, at this time, and I’ve made it to chapter eight when the concurrency discussion begins and you dive into Erlang’s message passing constructs.  I have to say that so far I am struggling with the language though.  I understand the concepts, it is just hard to apply them to real software.  The idea that you cannot have variables is really messing with my brain.  Once you assign a value to a variable it remains that way forever–everything is “const”.  To communicate with another process, send a message.  It is going to take some time to really get the ideas burnt into my brain.

Perhaps another issue is that the basic syntax is so different from the C-language syntax that is common amongst all of the programming languages I use:  C++, Perl, Obj-C, Assembler, and Verilog.  (Well I guess Assembler is different, but it ties right in their with C with the memory layout.)  I really need to dive in and solve a problem that I’ve solved in C++ so that I can see the bigger picture.  All of the talk about “funs” and pattern-matching and what not is not helping.  The section on dealing with binary data is mind blowing in its own way.  Bit vectors can be 9 bits in length!  I don’t think I’ve ever really considered anything but multiples of eight.  I just need to be patient and keep reading until I get to TCP sockets.

It has certainly been a very enlightening way to look at writing software thus far.  It is interesting to see a totally different approach to what I have grown accustomed to with OOP and imperative programming styles.  More to come in the future, I hope!