Class: Spacy::Language

Inherits:
Object
  • Object
show all
Defined in:
lib/ruby-spacy.rb

Overview

See also spaCy Python API document for Language.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(model = NO_MODEL, max_retrial: MAX_RETRIAL, timeout: 60, py_nlp: nil) ⇒ Language

Creates a language model instance, which is conventionally referred to by a variable named nlp.

Examples:

Load an installed spaCy model

nlp = Spacy::Language.new("en_core_web_sm")

Wrap an external pipeline (requires: pip install spacy-stanza)

py_nlp = PyCall.import_module("spacy_stanza").load_pipeline("ar")
nlp = Spacy::Language.new(py_nlp: py_nlp)

Parameters:

  • model (String) (defaults to: NO_MODEL) —

    A language model installed in the system

  • timeout (Numeric, nil) (defaults to: 60) —

    Seconds to wait for the model to load before raising a RuntimeError. nil waits indefinitely. The timeout is enforced on the Python side (a loading thread with join(timeout)) because Ruby's Timeout cannot fire while PyCall holds the GVL. When it fires, the loading thread is left running as a daemon until the process exits (accepted: timeouts are an abnormal path).

  • py_nlp (Object, nil) (defaults to: nil) —

    an existing Python Language pipeline to wrap instead of loading a model. For languages spaCy ships no trained pipeline for (e.g. Arabic and other right-to-left languages) or for self-built pipelines, create one with a third-party package such as spacy-stanza or spacy-udpipe and pass it here. Mutually exclusive with model; model name validation, timeout, and retries are skipped



677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
 
# File 'lib/ruby-spacy.rb', line 677

def initialize(model = NO_MODEL, max_retrial: MAX_RETRIAL, timeout: 60, py_nlp: nil)
  if py_nlp
    raise ArgumentError, "model and py_nlp: are mutually exclusive" unless model.equal?(NO_MODEL)
    unless Builtins.isinstance(py_nlp, PyLanguage)
      raise ArgumentError,
            "py_nlp: must be a spaCy Language pipeline " \
            "(e.g. from spacy.load or spacy_stanza.load_pipeline)"
    end

    @py_nlp = py_nlp
    return
  end

  model = "en_core_web_sm" if model.equal?(NO_MODEL)
  unless model.to_s.match?(/\A[a-zA-Z0-9_\-\.\/]+\z/)
    raise ArgumentError, "Invalid model name: #{model.inspect}"
  end

  retrial = 0
  begin
    @py_nlp = PyHelpers.load_with_timeout(model, timeout)
  rescue StandardError => e
    retrial += 1
    if retrial <= max_retrial
      sleep 0.5
      retry
    else
      raise "Failed to initialize Spacy after #{max_retrial} attempts: #{e.message}"
    end
  end
  # A timeout is not retried; it almost certainly means a hung load
  raise "PyCall execution timed out after #{timeout} seconds" if @py_nlp.nil?
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(name, *args) ⇒ Object

Methods defined in Python but not wrapped in ruby-spacy can be called by this dynamic method handling mechanism.



883
884
885
 
# File 'lib/ruby-spacy.rb', line 883

def method_missing(name, *args)
  Spacy.safe_py_send(@py_nlp, name, args)
end

Instance Attribute Details

#py_nlp ⇒ Object (readonly)

Returns a Python Language instance accessible via PyCall.

Returns:

  • (Object) —

    a Python Language instance accessible via PyCall



635
636
637
 
# File 'lib/ruby-spacy.rb', line 635

def py_nlp
  @py_nlp
end

Instance Method Details

#get_lexeme(text) ⇒ Object

A utility method to get a Python Lexeme object.

Parameters:

  • text (String) —

    A text string representing a lexeme

Returns:



751
752
753
 
# File 'lib/ruby-spacy.rb', line 751

def get_lexeme(text)
  @py_nlp.vocab[text]
end

#instance_variables_to_inspect ⇒ Object 



891
892
893
 
# File 'lib/ruby-spacy.rb', line 891

def instance_variables_to_inspect
  [:@spacy_nlp_id]
end

#matcher ⇒ Matcher

Generates a matcher for the current language model.

Returns:



719
720
721
 
# File 'lib/ruby-spacy.rb', line 719

def matcher
  Matcher.new(@py_nlp)
end

#memory_zone { ... } ⇒ Object

Executes a block within spaCy's memory zone for efficient memory management. Requires spaCy >= 3.8.

Yields:

  • the block to execute within the memory zone

Raises:

  • (NotImplementedError) —

    if spaCy version does not support memory zones



873
874
875
876
877
878
879
880
 
# File 'lib/ruby-spacy.rb', line 873

def memory_zone(&block)
  major, minor = SpacyVersion.split(".").map(&:to_i)
  unless major > 3 || (major == 3 && minor >= 8)
    raise NotImplementedError, "memory_zone requires spaCy >= 3.8 (current: #{SpacyVersion})"
  end

  PyCall.with(@py_nlp.memory_zone, &block)
end

#most_similar(vector, num) ⇒ Array<Hash{:key => Integer, :text => String, :best_rows => Array<Float>, :score => Float}>

Returns n lexemes having the vector representations that are the most similar to a given vector representation of a word.

Parameters:

  • vector (Object) —

    A vector representation of a word (whether existing or non-existing)

Returns:

  • (Array<Hash{:key => Integer, :text => String, :best_rows => Array<Float>, :score => Float}>) —

    An array of hash objects each contains the key, text, best_row and similarity score of a lexeme



765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
 
# File 'lib/ruby-spacy.rb', line 765

def most_similar(vector, num)
  vec_array = PyNp.asarray([vector])
  py_result = @py_nlp.vocab.vectors.most_similar(vec_array, n: num)
  key_texts = PyCall::List.call(PyHelpers.key_texts(@py_nlp, py_result[0][0].tolist))
  keys = key_texts.map { |kt| kt[0] }
  texts = key_texts.map { |kt| kt[1] }
  best_rows = PyCall::List.call(py_result[1])[0]
  scores = PyCall::List.call(py_result[2])[0]

  results = []
  num.times do |i|
    result = { key: keys[i].to_i,
               text: texts[i],
               best_row: best_rows[i],
               score: scores[i] }
    result.each_key do |key|
      result.define_singleton_method(key) { result[key] }
    end
    results << result
  end
  results
end

#phrase_matcher(attr: "ORTH") ⇒ PhraseMatcher

Generates a phrase matcher for the current language model. PhraseMatcher is more efficient than Matcher for matching large terminology lists.

Examples:

matcher = nlp.phrase_matcher(attr: "LOWER")
matcher.add("PRODUCT", ["iPhone", "MacBook Pro"])

Parameters:

  • attr (String) (defaults to: "ORTH") —

    the token attribute to match on (default: "ORTH"). Use "LOWER" for case-insensitive matching.

Returns:



731
732
733
 
# File 'lib/ruby-spacy.rb', line 731

def phrase_matcher(attr: "ORTH")
  PhraseMatcher.new(self, attr: attr)
end

#pipe(texts, disable: [], batch_size: 50) ⇒ Array<Doc>

Utility function to batch process many texts

Parameters:

  • texts (String)
  • disable (Array<String>) (defaults to: [])
  • batch_size (Integer) (defaults to: 50)

Returns:



793
794
795
796
797
 
# File 'lib/ruby-spacy.rb', line 793

def pipe(texts, disable: [], batch_size: 50)
  PyCall::List.call(@py_nlp.pipe(texts, disable: disable, batch_size: batch_size)).map do |py_doc|
    Doc.new(@py_nlp, py_doc: py_doc)
  end
end

#pipe_names ⇒ Array<String>

A utility method to list pipeline components.

Returns:

  • (Array<String>) —

    An array of text strings representing pipeline components



744
745
746
 
# File 'lib/ruby-spacy.rb', line 744

def pipe_names
  PyCall::List.call(@py_nlp.pipe_names).to_a
end

#read(text) ⇒ Object

Reads and analyze the given text.

Parameters:

  • text (String) —

    a text to be read and analyzed



713
714
715
 
# File 'lib/ruby-spacy.rb', line 713

def read(text)
  Doc.new(py_nlp, text: text)
end

#respond_to_missing?(sym, include_private = false) ⇒ Boolean

Returns:

  • (Boolean) 


887
888
889
 
# File 'lib/ruby-spacy.rb', line 887

def respond_to_missing?(sym, include_private = false)
  Spacy.py_hasattr?(@py_nlp, sym) || super
end

#spacy_nlp_id ⇒ String

Deprecated.

The Python object is no longer stored in a global variable at initialization time. Referencing this method creates a global variable in Python's __main__ on demand (which then stays alive until the process exits). Use #py_nlp instead.

Returns an identifier string that can be used to refer to the Python Language object inside PyCall::exec or PyCall::eval.

Returns:

  • (String) —

    an identifier string that can be used to refer to the Python Language object inside PyCall::exec or PyCall::eval



642
643
644
645
646
647
648
649
650
 
# File 'lib/ruby-spacy.rb', line 642

def spacy_nlp_id
  @spacy_nlp_id ||= begin
    warn "[DEPRECATION] `Spacy::Language#spacy_nlp_id` is deprecated. " \
         "It creates a Python global variable that is never released; use `py_nlp` instead."
    id = "nlp_#{@py_nlp.object_id}"
    Builtins.setattr(PyMain, id, @py_nlp)
    id
  end
end

#vocab(text) ⇒ Lexeme

Returns a ruby lexeme object

Parameters:

  • text (String) —

    a text string representing the vocabulary item

Returns:



758
759
760
 
# File 'lib/ruby-spacy.rb', line 758

def vocab(text)
  Lexeme.new(@py_nlp.vocab[text])
end

#vocab_string_lookup(id) ⇒ String

A utility method to lookup the string of the given vocabulary id.

Parameters:

  • id (Integer) —

    a vocabulary id (unsigned 64-bit values are supported)

Returns:

  • (String) —

    the string corresponding to the given vocabulary id



738
739
740
 
# File 'lib/ruby-spacy.rb', line 738

def vocab_string_lookup(id)
  PyHelpers.string_lookup(@py_nlp, Integer(id).to_s)
end

#with_llm(provider: :openai, **opts) {|OpenAIHelper, AnthropicHelper| ... } ⇒ Object

Yields a provider-specific LLM helper for making API calls within a block. The helper is configured once and reused for all calls within the block, making it efficient for batch processing with #pipe.

Providers:

Examples:

Claude

nlp.with_llm(provider: :anthropic) do |ai|
  ai.chat(system: "Analyze.", user: doc.linguistic_summary)
end

Local model via Ollama

nlp.with_llm(provider: :ollama, model: "llama3.2") do |ai|
  ai.chat(user: "Say hello.")
end

Parameters:

  • provider (Symbol) (defaults to: :openai) —

    :openai (default), :anthropic, or :ollama

  • opts (Hash) —

    helper options (access_token:, model:, max_tokens:, temperature:, base_url:, ...) — see OpenAIHelper#initialize and AnthropicHelper#initialize

Yields:

Returns:

  • (Object) —

    the block's return value



825
826
827
828
829
830
831
832
833
834
835
836
837
838
 
# File 'lib/ruby-spacy.rb', line 825

def with_llm(provider: :openai, **opts)
  helper = case provider.to_sym
           when :openai
             OpenAIHelper.new(**opts)
           when :anthropic
             AnthropicHelper.new(**opts)
           when :ollama
             OpenAIHelper.new(**{ base_url: "http://localhost:11434/v1",
                                  access_token: "ollama" }.merge(opts))
           else
             raise ArgumentError, "Unknown LLM provider: #{provider} (use :openai, :anthropic, or :ollama)"
           end
  yield helper
end

#with_openai(access_token: nil, model: OpenAIClient::DEFAULT_MODEL, max_completion_tokens: 1000, temperature: nil, base_url: nil) {|OpenAIHelper| ... } ⇒ Object

Yields an OpenAIHelper instance for making OpenAI API calls within a block. Equivalent to #with_llm with provider: :openai.

Examples:

Batch processing with pipe

nlp.with_openai(model: "gpt-5-mini") do |ai|
  nlp.pipe(texts).map do |doc|
    ai.chat(system: "Analyze.", user: doc.linguistic_summary)
  end
end

Parameters:

  • access_token (String, nil) (defaults to: nil) —

    OpenAI API key (defaults to OPENAI_API_KEY env var)

  • model (String) (defaults to: OpenAIClient::DEFAULT_MODEL) —

    the default model for chat requests

  • max_completion_tokens (Integer) (defaults to: 1000) —

    default maximum tokens in responses

  • temperature (Float, nil) (defaults to: nil) —

    default sampling temperature (omitted from requests when nil)

  • base_url (String, nil) (defaults to: nil) —

    OpenAI-compatible API endpoint override

Yields:

  • (OpenAIHelper) —

    the helper instance for making API calls

Returns:

  • (Object) —

    the block's return value



857
858
859
860
861
862
863
864
865
866
867
 
# File 'lib/ruby-spacy.rb', line 857

def with_openai(access_token: nil, model: OpenAIClient::DEFAULT_MODEL,
                max_completion_tokens: 1000, temperature: nil, base_url: nil)
  helper = OpenAIHelper.new(
    access_token: access_token,
    model: model,
    max_completion_tokens: max_completion_tokens,
    temperature: temperature,
    base_url: base_url
  )
  yield helper
end