bincache

BinCache is a system designed to chache compiled binaries on S3. It has resources for bash, ruby and chef.

BinCache tracks a single directory and a series of scripts. After each script is run, the contents of the directory is cached on s3. On sequential runs, the scripts are not re-executed, but instead the cache from S3 is downloaded and placed in the directory. If a script is changed, itself and all following scripts will be re-executed. BinCache does this by hashing the current script together with all previous scripts ran and storing the resulting cache in S3 with that hash as it’s name. By ensuring that scripts contain markers to introduce uniquness on different systems, you can make sure that you never have to wait twice for something to compile!

Install

## add rubygems.org to your gem sources list if it is not already there
gem sources -a https://rubygems.org

## install bincache
gem install bincache

Environment

Make sure you set the environment variables with something like:

## S3 access key id
export BINCACHE_S3_ACCESS_KEY=1234567890ABCDEFGHIJ

## S3 secret access key
export BINCACHE_S3_SECRET_KEY=1234567890ABCDEFHIJKLMNOPQRSTUVWXYZ12345

## S3 bucket
export BINCACHE_S3_BUCKET=my_ec2_bucket

## prefix inside bucket to store caches
export BINCACHE_S3_PREFIX=my_bincache/

Bash example

## you may have to put rubygem bins in your path by doing something like..
export PATH=$HOME/.gem/ruby/1.8/bin:$PATH

## set the directory to cache
dir=/tmp/bincache

## create a some sample scripts
cat <<-EOF > script1
  echo "I am in script one" > one
EOF

cat <<-EOF > script2
  echo "I am in script two" > two
EOF

## run bincache 
bincache $dir $script1 $script2

Ruby example

## make sure bincache is loaded
require 'bincache' 

## set the directory to cache
dir = "/tmp/bincache"

## create two sample scripts
script1 = <<EOS
  echo "I am in script one" > one
EOS

script2 = <<EOS
  echo "I am in script two" > two 
EOS

## create an empty list of scripts and add the sample script to it
scripts = []
scripts << script1
scripts << script2

## run bincache
bincache = BinCache.new
bincache.run_series(dir,scripts)

Chef example

require 'bincache'

bincache "script1" do
  action :run
  directory "/tmp/bincache"
  script <<-EOS
    echo "I am in script one" > one
  EOS
end

bincache "script2" do
  action :run
  directory "/tmp/bincache"
  script <<-EOS
    echo "I am in script two" > two
  EOS
end

uniquness example

To tag a script with something unique to ensure that it is distingushible on multiple systems, you need to insert something unique into the comments. For example, you could do:

echo "I am showing you how to uniqueify your script" > example
## #{`uname -m`}
## #{`lsb_release -ds`

When this script is hashed, it will produce different values on different systems. With this technique, you can cache different binaries from different systems with the same code base!