Class: TaskJuggler::TextParser::Scanner

Inherits:
Object
  • Object
show all
Defined in:
lib/taskjuggler/TextParser/Scanner.rb

Overview

The Scanner class is an abstract text scanner with support for nested include files and text macros. The tokenizer will operate on rules that must be provided by a derived class. The scanner is modal. Each mode operates only with the subset of token patterns that are assigned to the current mode. The current line is tracked accurately and can be used for error reporting. The scanner can operate on Strings or Files.

Direct Known Subclasses

ProjectFileScanner, RichTextScanner

Defined Under Namespace

Classes: BufferStreamHandle, FileStreamHandle, MacroStackEntry, StreamHandle

Instance Method Summary collapse

Constructor Details

#initialize(masterFile, log, tokenPatterns, defaultMode) ⇒ Scanner

Create a new instance of Scanner. masterFile must be a String that either contains the name of the file to start with or the text itself. messageHandler is a MessageHandler that is used for error messages. log is a Log to report progress and status.



209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 209

def initialize(masterFile, log, tokenPatterns, defaultMode)
  @masterFile = masterFile
  @messageHandler = TaskJuggler::MessageHandlerInstance.instance
  @log = log
  # This table contains all macros that may be expanded when found in the
  # text.
  @macroTable = MacroTable.new
  # The currently processed IO object.
  @cf = nil
  # This Array stores the currently processed nested files. It's an Array
  # of Arrays. The nested Array consists of 2 elements, the IO object and
  # the @tokenBuffer.
  @fileStack = []
  # This flag is set if we have reached the end of a file. Since we will
  # only know when the next new token is requested that the file is really
  # done now, we have to use this flag.
  @finishLastFile = false
  # True if the scanner operates on a buffer.
  @fileNameIsBuffer = false
  # A SourceFileInfo of the start of the currently processed token.
  @startOfToken = nil
  # Line number correction for error messages.
  @lineDelta = 0
  # Lists of regexps that describe the detectable tokens. The Arrays are
  # grouped by mode.
  @patternsByMode = { }
  # The currently active scanner mode.
  @scannerMode = nil
  # The mode that the scanner is in at the start and end of file
  @defaultMode = defaultMode
  # Points to the currently active pattern set as defined by the mode.
  @activePatterns = nil

  tokenPatterns.each do |pat|
    type = pat[0]
    regExp = pat[1]
    mode = pat[2] || :tjp
    postProc = pat[3]
    addPattern(type, regExp, mode, postProc)
  end
  self.mode = defaultMode
end

Instance Method Details

#addMacro(macro) ⇒ Object

Add a Macro to the macro translation table.



435
436
437
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 435

def addMacro(macro)
  @macroTable.add(macro)
end

#addPattern(type, regExp, mode, postProc = nil) ⇒ Object

Add a new pattern to the scanner. type is either nil for tokens that will be ignored, or some identifier that will be returned with each token of this type. regExp is the RegExp that describes the token. mode identifies the scanner mode where the pattern is active. If it’s only a single mode, mode specifies the mode directly. For multiple modes, it’s an Array of modes. postProc is a method reference. This method is called after the token has been detected. The method gets the type and the matching String and returns them again in an Array.



260
261
262
263
264
265
266
267
268
269
270
271
272
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 260

def addPattern(type, regExp, mode, postProc = nil)
  if mode.is_a?(Array)
    mode.each do |m|
      # The pattern is active in multiple modes
      @patternsByMode[m] = [] unless @patternsByMode.include?(m)
      @patternsByMode[m] << [ type, regExp, postProc ]
    end
  else
    # The pattern is only active in one specific mode.
    @patternsByMode[mode] = [] unless @patternsByMode.include?(mode)
    @patternsByMode[mode] << [ type, regExp, postProc ]
  end
end

#close ⇒ Object

Finish processing and reset all data structures.



304
305
306
307
308
309
310
311
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 304

def close
  unless @fileNameIsBuffer
    @log.startProgressMeter("Reading file #{@masterFile}")
    @log.stopProgressMeter
  end
  @fileStack = []
  @cf = @tokenBuffer = nil
end

#columnNo ⇒ Object

:nodoc:



369
370
371
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 369

def columnNo # :nodoc:
  0
end

#error(id, text, sfi = nil, data = nil) ⇒ Object

Call this function to report any errors related to the parsed input.



466
467
468
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 466

def error(id, text, sfi = nil, data = nil)
  message(:error, id, text, sfi, data)
end

#expandMacro(prefix, args, callLength) ⇒ Object

Expand a macro and inject it into the input stream. prefix is any string that was found right before the macro call. We have to inject it before the expanded macro. args is an Array of Strings. The first is the macro name, the rest are the parameters. callLength is the number of characters for the complete macro call “$…”.



449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 449

def expandMacro(prefix, args, callLength)
  # Get the expanded macro from the @macroTable.
  macro, text = @macroTable.resolve(args, sourceFileInfo)

  # If the expanded macro is empty, we can ignore it.
  return if text == ''

  unless macro && text
    error('undefined_macro', "Undefined macro '#{args[0]}' called")
  end

  unless @cf.injectMacro(macro, args, prefix + text, callLength)
    error('macro_stack_overflow', "Too many nested macro calls.")
  end
end

#fileName ⇒ Object

Return the name of the currently processed file. If we are working on a text buffer, the text will be returned.



361
362
363
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 361

def fileName
  @cf ? @cf.fileName : @masterFile
end

#include(includeFileName, sfi, &block) ⇒ Object

Continue processing with a new file specified by includeFileName. When this file is finished, we will continue in the old file after the location where we started with the new file. The method returns the full qualified name of the included file.



317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 317

def include(includeFileName, sfi, &block)
  if includeFileName[0] != '/'
    pathOfCallingFile = @fileStack.last[0].dirname
    path = pathOfCallingFile.empty? ? '' : pathOfCallingFile + '/'
    # If the included file is not an absolute name, we interpret the file
    # name relative to the including file.
    includeFileName = path + includeFileName
  end

  # Try to dectect recursive inclusions. This will not work if files are
  # accessed via filesystem links.
  @fileStack.each do |entry|
    if includeFileName == entry[0].fileName
      error('include_recursion',
            "Recursive inclusion of #{includeFileName} detected", sfi)
    end
  end

  # Save @tokenBuffer in the record of the parent file.
  @fileStack.last[1] = @tokenBuffer unless @fileStack.empty?
  @tokenBuffer = nil
  @finishLastFile = false

  # Open the new file and push the handle on the @fileStack.
  begin
    @fileStack << [ (@cf = FileStreamHandle.new(includeFileName, @log,
                                                self)), nil, block ]
    @log.msg { "Parsing file #{includeFileName}" }
  rescue StandardError
    error('bad_include', "Cannot open include file #{includeFileName}", sfi)
  end

  # Return the name of the included file.
  includeFileName
end

#line ⇒ Object

:nodoc:



373
374
375
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 373

def line # :nodoc:
  @cf ? @cf.line : 0
end

#lineNo ⇒ Object

:nodoc:



365
366
367
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 365

def lineNo # :nodoc:
  @cf ? @cf.lineNo : 0
end

#macroDefined?(name) ⇒ Boolean

Return true if the Macro name has been added already.

Returns:

  • (Boolean) 


440
441
442
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 440

def macroDefined?(name)
  @macroTable.include?(name)
end

#mode=(newMode) ⇒ Object

Switch the parser to another mode. The scanner will then only detect patterns of that newMode.



276
277
278
279
280
281
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 276

def mode=(newMode)
  #puts "**** New mode: #{newMode}"
  @activePatterns = @patternsByMode[newMode]
  raise "Undefined mode #{newMode}" unless @activePatterns
  @scannerMode = newMode
end

#nextToken ⇒ Object

Return the next token from the input stream. The result is an Array with 3 entries: the token type, the token String and the SourceFileInfo where the token started.



380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 380

def nextToken
  # If we have a pushed-back token, return that first.
  unless @tokenBuffer.nil?
    res = @tokenBuffer
    @tokenBuffer = nil
    return res
  end

  if @finishLastFile
    # The previously processed file has now really been processed to
    # completion. Close it and remove the corresponding entry from the
    # @fileStack.
    @finishLastFile = false
    #@log.msg { "Completed file #{@cf.fileName}" }

    # If we have a block to be executed on EOF, we call it now.
    onEof = @fileStack.last[2]
    onEof.call if onEof

    @cf.close if @cf
    @fileStack.pop

    if @fileStack.empty?
      # We are done with the top-level file now.
      @cf = @tokenBuffer = nil
      @finishLastFile = true
      return [ :endOfText, '<EOT>', @startOfToken ]
    else
      # Continue parsing the file that included the current file.
      @cf, tokenBuffer = @fileStack.last
      @log.msg { "Parsing file #{@cf.fileName} ..." }
      # If we have a left over token from previously processing this file,
      # return it now.
      if tokenBuffer
        @finishLastFile = true if tokenBuffer[0] == :eof
        return tokenBuffer
      end
    end
  end

  scanToken
end

#open(fileNameIsBuffer = false) ⇒ Object

Start the processing. If fileNameIsBuffer is true, we operate on a String, else on a File.



286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 286

def open(fileNameIsBuffer = false)
  @fileNameIsBuffer = fileNameIsBuffer
  if fileNameIsBuffer
    @fileStack = [ [ @cf = BufferStreamHandle.new(@masterFile, @log, self),
                     nil, nil ] ]
  else
    begin
      @fileStack = [ [ @cf = FileStreamHandle.new(@masterFile, @log, self),
                       nil, nil ] ]
    rescue IOError, SystemCallError
      error('open_file', "Cannot open file #{@masterFile}: #{$!}")
    end
  end
  @masterPath = @cf.dirname + '/'
  @tokenBuffer = nil
end

#returnToken(token) ⇒ Object

Return a token to retrieve it with the next nextToken() call again. Only 1 token can be returned before the next nextToken() call.



425
426
427
428
429
430
431
432
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 425

def returnToken(token)
  #@log.msg { "-> Returning Token: [#{token[0]}][#{token[1]}]" }
  unless @tokenBuffer.nil?
    $stderr.puts @tokenBuffer
    raise "Fatal Error: Cannot return more than 1 token in a row"
  end
  @tokenBuffer = token
end

#sourceFileInfo ⇒ Object

Return SourceFileInfo for the current processing prosition.



354
355
356
357
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 354

def sourceFileInfo
  @cf ? SourceFileInfo.new(fileName, @cf.lineNo - @lineDelta, 0) :
        SourceFileInfo.new(@masterFile, 0, 0)
end

#warning(id, text, sfi = nil, data = nil) ⇒ Object 



470
471
472
 
# File 'lib/taskjuggler/TextParser/Scanner.rb', line 470

def warning(id, text, sfi = nil, data = nil)
  message(:warning, id, text, sfi, data)
end