Class: SelBackup

Inherits:
Object
  • Object
show all
Defined in:
lib/selbackup.rb

Overview

SelBackup Class

Enables connection to s3 server and uploads files

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(key, secret, bucket, region = 'eu-west-1') ⇒ SelBackup

Public: Initialize a new SelBackup instance.

key

A String representing the AWS ACCESS KEY ID.

secret

A String representing the AWS ACCESS KEY SECRET.

bucket

A String representing the name of the bucket ot use.

:region (optional)

A String representing the region to conect to (default: ‘eu-west-1’).

Examples

SelectraBackup.new('key', 'secret', 'bucket', region: 'us-east-1')

Returns the newly instanciated object.



29
30
31
32
33
34
 
# File 'lib/selbackup.rb', line 29

def initialize(key, secret, bucket, region = 'eu-west-1')
  @access_key = key
  @access_secret = secret
  @bucket_name = bucket
  @region = region
end

Instance Attribute Details

#access_key ⇒ Object

Attributes: informations required for connection to s3 server



10
11
12
 
# File 'lib/selbackup.rb', line 10

def access_key
  @access_key
end

#access_secret ⇒ Object

Attributes: informations required for connection to s3 server



10
11
12
 
# File 'lib/selbackup.rb', line 10

def access_secret
  @access_secret
end

#bucket_name ⇒ Object

Attributes: informations required for connection to s3 server



10
11
12
 
# File 'lib/selbackup.rb', line 10

def bucket_name
  @bucket_name
end

#region ⇒ Object

Attributes: informations required for connection to s3 server



10
11
12
 
# File 'lib/selbackup.rb', line 10

def region
  @region
end

Instance Method Details

#bucket ⇒ Object

Public: Returns the S3 bucket.

Returns a Fog::Storage::AWS::Directory instance.



40
41
42
 
# File 'lib/selbackup.rb', line 40

def bucket
  @bucket ||= connection.directories.get(bucket_name)
end

#check_daily(filename, nb_day) ⇒ Object

Public: daily check algorythm

filename

String which contains a filename. If you upload ‘2013-01-01-file.tgz’, filename will be ‘file.tgz’

  • If 7 daily files or less: does nothing

  • If more than 7 daily files but less than one week datediff between the latest daily file and the most recent weekly file: deletes the latest daily file

  • If more than 7 daily files but less than one week datediff between the latest daily file and the most recent weekly file: pushes the latest daily file to weekly file

    2013-01-01-daily-file.tgz -> 2013-01-01-weekly-file.tgz

Returns true if a file as been push. False in other cases



57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/selbackup.rb', line 57

def check_daily(filename, nb_day)
  return_value = false
  daily_directory = give_me_files(filename, "daily")
  if ! daily_good?(daily_directory, nb_day)
    weekly_directory = give_me_files(filename, "weekly")
    latest_daily_file = give_me_latest_file(daily_directory)
    recent_weekly_file = give_me_recent_file(weekly_directory)
    if weekly_directory == [] || should_weekly?(latest_daily_file.key, recent_weekly_file.key)
      weekly_file = gen_weekly_file({ key: latest_daily_file.key, body: latest_daily_file.body})
      push_file(weekly_file)
      return_value = true
    end
    delete_file(latest_daily_file)
  end
  return return_value
end

#check_monthly(filename, nb_month) ⇒ Object

Public: montlhy check algorythm

filename

String which contains a filename. If you upload ‘2013-01-01-file.tgz’, filename will be ‘file.tgz’

  • If 3 monthly files or less: does nothing

  • if more than 3 monthly files: deletes the latest monthly file

Returns true if a file as been delete. False in other cases



84
85
86
87
88
89
90
91
92
93
 
# File 'lib/selbackup.rb', line 84

def check_monthly(filename, nb_month)
  return_value = false
  monthly_directory = give_me_files(filename, "monthly")
  if ! monthly_good?(monthly_directory, nb_month)
    latest_monthly_file = give_me_latest_file(monthly_directory)
    delete_file(latest_monthly_file)
    return_value = true
  end
  return return_value
end

#check_weekly(filename, nb_week) ⇒ Object

Public: weekly check algorythm

filename

String which contains a filename. If you upload ‘2013-01-01-file.tgz’, filename will be ‘file.tgz’

  • If 4 weekly files or less: does nothing

  • If more than 4 weekly files but less than one month datediff between the latest weekly file and the most recent monthly file: deletes the latest weekly file

  • If more than 4 weekly files but less than one week datediff between the latest weekly file and the most recent monthly file: pushes the latest weekly file to monthly file

    2013-01-01-weekly-file.tgz -> 2013-01-01-monthly-file.tgz

Returns true if a file as been push. False in other cases



108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
 
# File 'lib/selbackup.rb', line 108

def check_weekly(filename, nb_week)
  return_value = false
  weekly_directory = give_me_files(filename, "weekly")
  if ! weekly_good?(weekly_directory, nb_week)
    monthly_directory = give_me_files(filename, "monthly")
    latest_weekly_file = give_me_latest_file(weekly_directory)
    recent_monthly_file = give_me_recent_file(monthly_directory)
    if monthly_directory == [] || should_monthly?(latest_weekly_file.key, recent_monthly_file.key)
      monthly_file = gen_monthly_file({ key: latest_weekly_file.key, body: latest_weekly_file.body})
      push_file(monthly_file)
      return_value = true
    end
    delete_file(latest_weekly_file)
  end
  return return_value
end

#connection ⇒ Object

Public: Returns the S3 connection.

Returns a Fog::Storage::AWS::Real instance.



129
130
131
 
# File 'lib/selbackup.rb', line 129

def connection
  @connection ||= Fog::Storage.new(provider: 'AWS', aws_access_key_id: access_key, aws_secret_access_key: access_secret, region: region)
end

#daily_good?(directoryFiles, nb_day) ⇒ Boolean

Public: Check daily files number

directoryFiles

Array of daily files

['file/2013-01-01-daily-file.tgz', 'file/2013-01-02-daily-file.tgz', ...]

Returns false if there is more than 7 daily files. False in other cases

Returns:

  • (Boolean) 


140
141
142
 
# File 'lib/selbackup.rb', line 140

def daily_good?(directoryFiles, nb_day)
  return directoryFiles.length > nb_day ? false : true;
end

#date_from_filename(filename) ⇒ Object

Public: Returns the parsed date.

filename: “2016-11-26-daily-server_name-project1234.pgsql.gz” for example

Returns a Date instance.



187
188
189
 
# File 'lib/selbackup.rb', line 187

def date_from_filename(filename)
  Date.parse(filename.match(/\d{4}-\d{2}-\d{2}/).to_s)
end

#delete_file(file) ⇒ Object

Public: deletes a file

file

Fog::Storage::AWS::Files file

returns nothing



150
151
152
 
# File 'lib/selbackup.rb', line 150

def delete_file(file)
  file.destroy
end

#directory(path) ⇒ Object

Public: Returns a virtual S3 directory.

S3 doesn’t know such a thing as a directory but it is possible to use a prefix to select only the files having a key starting with this prefix.

path

A String representing the prefix to use.

Examples

sb = SelectraBackup.new('key', 'secret', 'bucket')
backup = sb.directory('backup/')
backup.files
# => <Fog::Storage::AWS::Files ...>

Returns a Fog::Storage::AWS::Directory instance.



169
170
171
 
# File 'lib/selbackup.rb', line 169

def directory(path)
  connection.directories.get(bucket_name, prefix: path)
end

#files ⇒ Object

Public: Returns the list of files.

Returns a Fog::Storage::AWS::Files instances.



177
178
179
 
# File 'lib/selbackup.rb', line 177

def files
  bucket.files
end

#gen_daily_file(file) ⇒ Object

Public: gen a daily file

file

file you want to upload

Returns an hash containing the filename, the key and the body of the daily file



260
261
262
263
264
265
266
267
268
269
270
271
 
# File 'lib/selbackup.rb', line 260

def gen_daily_file(file)
  name = File.basename(file)
  if name =~ /[0-9]{4}-[0-9]{2}-[0-9]{2}(\.\w*)?$/
    return false
  elsif !(name =~ /[0-9]{4}-[0-9]{2}-[0-9]{2}-/)
    name = "#{Date.today.to_s}-#{name}"
  end
  myname = name.split(/[0-9]{4}-[0-9]{2}-[0-9]{2}-/).last
  mydate = name.split("-#{myname}").first
  key = "#{myname}/#{mydate}-daily-#{myname}"
  return { myname: myname, key: key, body: File.open(file) }
end

#gen_monthly_file(file) ⇒ Object

Public: gen a monthly file

file

hash containing the key and the body of the weekly file you want to push to monthly file

Returns an hash containing the key and the body of the monthly file



279
280
281
282
283
284
285
286
 
# File 'lib/selbackup.rb', line 279

def gen_monthly_file(file)
  if file[:key] =~ /[\w+.-]+\/[\w+.-]+-\d{4}-\d{2}-\d{2}-[\w+.-]+/
    monthly_name = file[:key].sub("/weekly-", "/monthy-")
  else
    monthly_name = file[:key].sub("-weekly-", "-monthly-")
  end
  return { key: monthly_name, body: file[:body] }
end

#gen_weekly_file(file) ⇒ Object

Public: gen a weekly file

file

hash containing the key and the body of the daily file you want to push to weekly file

Returns an hash containing the key and the body of the weekly file



294
295
296
297
298
299
300
301
 
# File 'lib/selbackup.rb', line 294

def gen_weekly_file(file)
  if file[:key] =~ /[\w+.-]+\/[\w+.-]+-\d{4}-\d{2}-\d{2}-[\w+.-]+/
    weekly_name = file[:key].sub("/daily-", "/weekly-")
  else
    weekly_name = file[:key].sub("-daily-", "-weekly-")
  end
  return { key: weekly_name, body: file[:body] }
end

#give_me_files(filename, type) ⇒ Object

Public: returns lists of file

filename: “file.tgz” for example

type: specify if you want daily, weekly or monthly files

Returns a sorted array



199
200
201
202
203
204
205
206
207
208
209
 
# File 'lib/selbackup.rb', line 199

def give_me_files(filename, type)
  full_directory = directory("#{filename}/")
  files = []
  full_directory.files.each do | file |
    if file.key =~ /#{filename}\/\d{4}-\d{2}-\d{2}-#{type}-[\w+.]+/ ||
       file.key =~ /#{filename}\/#{type}-\d{4}-\d{2}-\d{2}-[\w+.]+/
        files.push(file)
    end
  end
  return files
end

#give_me_latest_file(files) ⇒ Object

Public: returns latest file

files: array of AWS object file

Returns the latest file, based on the date



217
218
219
220
221
222
223
224
225
 
# File 'lib/selbackup.rb', line 217

def give_me_latest_file(files)
  latest_file = files.first
  files.each do | file |
    if date_from_filename(file.key) < date_from_filename(latest_file.key)
      latest_file = file
    end
  end
  return latest_file
end

#give_me_recent_file(files) ⇒ Object

Public: returns most recent file

files: array of AWS object file

Returns the most recent file, based on the date



233
234
235
236
237
238
239
240
241
 
# File 'lib/selbackup.rb', line 233

def give_me_recent_file(files)
  recent_file = files.first
  files.each do | file |
    if date_from_filename(file.key) > date_from_filename(recent_file.key)
      recent_file = file
    end
  end
  return recent_file
end

#monthly_good?(files, nb_month) ⇒ Boolean

Public: Check monthly files number

files

Array of monthly files

['file/2013-01-01-monthly-file.tgz', 'file/2013-01-02-monthly-file.tgz', ...]

Returns false if there is more than 3 monthly files. False in other cases

Returns:

  • (Boolean) 


250
251
252
 
# File 'lib/selbackup.rb', line 250

def monthly_good?(files, nb_month)
  return files.length > nb_month ? false : true
end

#push_file(file) ⇒ Object

Public: pushes a file to s3 server

file

hash containing the key and the body of the file you want to upload

Returns nothing



309
310
311
312
313
314
315
316
317
 
# File 'lib/selbackup.rb', line 309

def push_file(file)
  if file[:key] =~ /[\w+.-]+\/[\w+.-]+-\d{4}-\d{2}-\d{2}-[\w+.-]+/
    filename = file[:key].split('/').first
    name_parts = file[:key].split('/').last.split('-')
    file[:key] = "#{filename}/#{name_parts[1]}-#{name_parts[2]}-#{name_parts[3]}-#{name_parts[0]}-#{filename}"
  end
  # 104857600 bytes => 100 megabytes
  bucket.files.create(key: file[:key], body: file[:body], multipart_chunk_size: 104857600)
end

#should_monthly?(file1, file2) ⇒ Boolean

Public: date diff

file1

latest weekly file

file2

most recent monthly file

Returns true if datediff between the two files is upper than 1 month. False in other cases

Returns:

  • (Boolean) 


347
348
349
350
351
352
353
354
355
356
 
# File 'lib/selbackup.rb', line 347

def should_monthly?(file1, file2)
  begin
    date1 = date_from_filename(file1).prev_month
    date2 = date_from_filename(file2)
  rescue
    print "Wrong date (Date.parse in should_monthly)."
    return false
  end
  return date1 >= date2 ? true : false
end

#should_weekly?(file1, file2) ⇒ Boolean

Public: date diff

file1

latest daily file

file2

most recent weekly file

Returns true if datediff between the two files is upper than 7 days. False in other cases

Returns:

  • (Boolean) 


327
328
329
330
331
332
333
334
335
336
337
 
# File 'lib/selbackup.rb', line 327

def should_weekly?(file1, file2)
  begin
    date1 = date_from_filename(file1)
    date2 = date_from_filename(file2)
  rescue
    print "Wrong date (Date.parse in should_monthly)."
    return false
  end
  date_diff = date1 - date2
  return date_diff.abs >= 7 ? true : false
end

#upload_file(file, nb_day = 7, nb_week = 4, nb_month = 3) ⇒ Object

Public: upload file algorythm

file

file you want to upload

Uploads file as daily file and check daily, weekly and monthly full.

Returns nothing



365
366
367
368
369
370
371
372
373
374
375
376
 
# File 'lib/selbackup.rb', line 365

def upload_file(file, nb_day = 7, nb_week = 4, nb_month = 3)
  daily_file = gen_daily_file(file)
  if daily_file == false
    return false
  end
  push_file(daily_file)
  if check_daily(daily_file[:myname], nb_day)
    if check_weekly(daily_file[:myname], nb_week)
      check_monthly(daily_file[:myname], nb_month)
    end
  end
end

#weekly_good?(directoryFiles, nb_week) ⇒ Boolean

Public: Check weekly files number

files

Array of daily files

['file/2013-01-01-weekly-file.tgz', 'file/013-01-02-weekly-2file.tgz', ...]

Returns false if there is more than 4 weekly files. False in other cases

Returns:

  • (Boolean) 


385
386
387
 
# File 'lib/selbackup.rb', line 385

def weekly_good?(directoryFiles, nb_week)
  return directoryFiles.length > nb_week ? false : true
end