Module: TestRailRSpecIntegration

Defined in:
lib/files/testrail_rspec_integration.rb

Defined Under Namespace

Classes: TestRailPlanFormatter

Constant Summary collapse

@@total_count =

For pushing results up to an existing test run in TestRail, no matter whether the test run is independent or grouped under a test plan # This is different from simply creating a stand-alone test run from the results of the test. The tricky part about this is Jenkins run rspecs on multiple processes with different batches of rspec tests.

0
@@run_count = 
0
@@skip_count = 
0

Class Method Summary collapse

Class Method Details

.add_formatter_for(config) ⇒ Object

Adds a documentation formatter to the rspec if one is not there already.



229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
 
# File 'lib/files/testrail_rspec_integration.rb', line 229

def self.add_formatter_for(config)
  # For some reason, adding a custom formatter will remove any other formatters.
  # Thus during execution nothing gets printed to the screen. Need to add another
  # formatter to indicate some sort of progress
  found_doc_formatter = false
  config.formatters.each do |fm|
    if (fm.class == RSpec::Core::Formatters::DocumentationFormatter)
      found_doc_formatter = true
      break
    end
  end
  unless found_doc_formatter
    config.add_formatter "doc"
  end
end

.add_rspec_callback(config, product, add_formatter: true) ⇒ Object

Registers a callback custom formatter to an rspec. The new test run is created from the results of the tests. This is in effect the opposite of the method above (register_rspec_integration).



420
421
422
423
424
425
426
 
# File 'lib/files/testrail_rspec_integration.rb', line 420

def self.add_rspec_callback(config, product, add_formatter: true)
  config.add_formatter TestRailRSpecIntegration::TestRailPlanFormatter
  TestRailRSpecIntegration::TestRailPlanFormatter.set_product(product)
  if add_formatter
    TestRailRSpecIntegration.add_formatter_for(config)
  end
end

.filter_rspecs_by_test_run(config, test_run_cases) ⇒ Object

Filters an rspec run by testrail_id’s for bridge Filters an rspec run by testcases found in a particular testrun on testrail.



282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
 
# File 'lib/files/testrail_rspec_integration.rb', line 282

def self.filter_rspecs_by_test_run(config, test_run_cases)
  # This lambda gets called once for each example
  # Here value is an array of string test case ID's.
  config.filter_run_including testrail_id: lambda { |value|
    @@total_count += 1
    unless value.is_a? Array
      @@skip_count += 1
      puts "ERROR! testcase has invalid testrail ID: #{value}. Value should be an array, got: #{value.class}".red
      return false
    end
    # The test id's are strings. Convert them to integers to make comparison easier
    test_ids = value.collect { |str| str.to_i }
    # Compute the intersection using the handy &() method
    intersect = test_run_cases.keys & test_ids
    # Do not include if the test cases have already been run and have ALL passed.
    # (That would be a waste of time to rerun test's that have already passed)
    pass_count = 0
    skip_count = 0
    # Do include if the intersection contains a test id
    if intersect.size > 0
      test_ids.each do |id|
        test_case = test_run_cases[id]
        if test_case.nil?
          next
        end
        # puts "   #{id} temp id: #{test_case.temp_id} Status: #{test_case.status}, "
        pass_count += 1 if test_case.status == :passed
        skip_count += 1 if test_case.status == :pending
      end
      all_passed = pass_count == test_ids.count
      all_skipped = skip_count == test_ids.count
      if all_passed
        @@skip_count += 1
        puts "Skipping test case #{value}, because all tests already passed"
      end
      if all_skipped
        @@skip_count += 1
        puts "Skipping test case #{value}, because all tests marked pending"
      end
      do_execute = (pass_count + skip_count) != test_ids.count
      @@run_count += 1 if do_execute
      do_execute
    else
      @@skip_count += 1
      false
    end
  }
end

.filter_rspecs_by_test_run_and_user(config, user_id, test_run_cases) ⇒ Object

Takes care of filtering out tests that are NOT assigned to the user. So essentially runs only tests specified in a testrun in testrail, and that are assigned to a particular user. config - The Rspec configuration user_id - An integer ID corresponding to the testrail user test_run_cases - A hash of TestCase instances



250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
 
# File 'lib/files/testrail_rspec_integration.rb', line 250

def self.filter_rspecs_by_test_run_and_user(config, user_id, test_run_cases)
  config.filter_run_including testrail_id: lambda { |value|
    # The test id's are strings. Convert them to integers to make comparison easier
    test_ids = value.collect { |str| str.to_i }
    # Compute the intersection using the handy &() method
    intersect = test_run_cases.keys & test_ids
    assigned_to_ids = []
    # Do include if the intersection contains a test id
    if intersect.size > 0
      test_ids.each do |id|
        test_case = test_run_cases[id]
        if test_case.nil?
          next
        end
        assigned_to_ids << test_case.assigned_to
      end
      # return true to execute the test if any one of the testcase ID's is assigned to the user
      do_execute = assigned_to_ids.include? user_id
      if do_execute
        puts "Assigned to user. Including testcase ID's: #{value}"
      else
        puts "Not assigned to user: Skipping #{value}"
      end
      do_execute
    else
      false
    end
  }
end

.filter_rspecs_by_testid(config, test_run_cases) ⇒ Object

Filters an rspec run by test_id’s for canvas. This is used for filtering out test cases that have already been run previously, say on a previous test run that was aborted early and restarted. In this case we skip tests that already passed or were marked as pending (rspec for skipped)



335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
 
# File 'lib/files/testrail_rspec_integration.rb', line 335

def self.filter_rspecs_by_testid(config, test_run_cases)
  # This lambda gets called once for each example
  # Here value is an array of string test case ID's.
  config.filter_run_including test_id: lambda { |id|
    @@total_count += 1
    id = id.to_i
    # The test id's are integers, and in canvas there is only one ID per test case, NOT an array like Bridge
    in_run = test_run_cases.keys.include?( id )

    # Do include if the intersection contains a test id
    if in_run
      test_case = test_run_cases[id]

      if (test_case.status == :passed)
        @@skip_count += 1
        puts "Skipping test case #{id}, because it has already passed"
        return false
      end

      if (test_case.status == :pending)
        @@skip_count += 1
        puts "Skipping test case #{id}, because it is marked pending"
        return false
      end

      @@run_count += 1
      return true # do execute this test
    else
      @@skip_count += 1
	puts "Skipping test case #{id}, because it was not in test_run_cases"
      return false
    end
  }
end

.get_run_count ⇒ Object 



224
225
226
 
# File 'lib/files/testrail_rspec_integration.rb', line 224

def self.get_run_count
  @@run_count
end

.get_skip_count ⇒ Object 



220
221
222
 
# File 'lib/files/testrail_rspec_integration.rb', line 220

def self.get_skip_count
  @@skip_count
end

.get_total_count ⇒ Object 



216
217
218
 
# File 'lib/files/testrail_rspec_integration.rb', line 216

def self.get_total_count
  @@total_count
end

.register_rspec_integration(config, product, add_formatter: true) ⇒ Object

The param is an RSPEC config The second param is a symbol for which product to hook into



372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
 
# File 'lib/files/testrail_rspec_integration.rb', line 372

def self.register_rspec_integration(config, product, add_formatter: true)
  # Runs test cases as found in a test run on testrail

  # This will select test examples to run based off of what test rail defines, not what
  # the file pattern on the command line defines.
  # That is, this will take a test run (int test rail), and run all the cases defined in it.

  # First clear any filters passed in from the command line
  config.inclusion_filter = nil
  test_run_cases = TestRailOperations.get_test_run_cases(ENV["TESTRAIL_RUN_ID"].to_i)

  user_id = nil
  unless ENV["TESTRAIL_ASSIGNED_TO"].nil?
    user_json = TestRailOperations.get_test_rail_user_by_email(ENV["TESTRAIL_ASSIGNED_TO"])
    user_id = user_json["id"]
    puts "Testrail assigned to: #{user_json}"
  end

  if user_id
    TestRailRSpecIntegration.filter_rspecs_by_test_run_and_user(config, user_id, test_run_cases)
  else
    case(product)
    when :bridge
      TestRailRSpecIntegration.filter_rspecs_by_test_run(config, test_run_cases)
    when :canvas
      TestRailRSpecIntegration.filter_rspecs_by_testid(config, test_run_cases)
    end
  end

  config.add_formatter TestRailRSpecIntegration::TestRailPlanFormatter
  TestRailRSpecIntegration::TestRailPlanFormatter.set_product(product)
  if add_formatter
    TestRailRSpecIntegration.add_formatter_for(config)
  end

  # Captures and posts results for any remaining test case results in @@cases that don't fill a full batch
  config.after(:suite) do |suite|
    total_cases = TestRailPlanFormatter.class_variable_get(:@@cases)

    if total_cases.size > 0
      TestRailRSpecIntegration::TestRailPlanFormatter.post_results total_cases
    end
  end
end