Getting started with the CASS tools

CASS (Contrast Analysis of Semantic Similarity) is a set of tools for conducting contrast-based analyses of semantic similarity in text. CASS is based on the BEAGLE model described by Jones and Mewhort (2007). For a more detailed explanation, see Holtzman et al (under review).

License

Installation

The CASS tools are packaged as a library for the Ruby programming language. You must have Ruby interpreter installed on your system, as well as the NMatrix library. To install, follow these steps:

(1) Install Ruby–preferably 1.9 or greater. Installers for most platforms are available here.

(2) Install the NArray[http://narray.rubyforge.org] library. On most platforms, you should be able to just type:

	gem install narray

On Windows, it’s a bit more involved; follow the instructions here.

(3) Install the CASS gem from the command prompt, like so:

gem install cass

(4) Download the sample analysis files (cass_sample.zip), which will help you get started working with CASS. Unpack the file anywhere you like, and you should be ready to roll.

Usage

There are two general ways to use CASS:

The easy way

For users without prior programming experience, CASS is streamlined to make things as user-friendly as possible. Assuming you’ve installed CASS per the instructions above, you can run a full CASS analysis just by tweaking a few settings and running the analysis script (run_cass.rb) included in the sample analysis package (cass_sample.zip). Detailed instructions are available in [this] tutorial; in brief, here’s what you need to do to get up and running:

Download cass_sample.zip and unpack it somewhere. The package contains several files that tell CASS what to do, as well as some sample text you can process. These include:

contrasts.txt: a specification of the contrasts you’d like to run on the text, one per line. For a detailed explanation of the format, see the [tutorial].
default.spec: the main specification file containing all the key settings CASS needs in order to run. All settings are commented, but a more detailed explanation is provided in the [tutorial]. You can create as many .spec files as you like (no need to edit this one repeatedly!), just make sure to edit run_cass.rb to indicate which .spec file to use.
stopwords.txt: A sample list of stopwords to exclude from analysis (CASS will use this file by default). These are mostly high-frequency function words that carry little meaning but can strongly bias a text.
sample1.txt and sample2.txt: two sample documents to get you started.
run_cass.rb: the script you’ll use to run CASS.

Edit the settings as you please. An explanation of what everything means is in the [tutorial]; if you’re just getting started, you can just leave everything as is and run the sample analysis.
Run run_cass.rb. If you’re on Windows, you may be able to double-click on the script to run it; however, if you do that, you won’t see any of the output. On most platforms (and optionally on Windows), you’ll have to run the script from the command prompt. You can do this by opening up a terminal window (or, in Windows, a command prompt), navigating to the directory that contains the sample analysis files, and typing:

ruby run_cass.rb After doing that, you should get a bunch of output showing you exactly what’s going on. There should also be some new files in the working directory containing the results of the analysis.

Assuming the analysis ran successfully, you can now set about running your own analyses. We recommend reading the entire [tutorial] before diving in.

As a library

Advanced users familiar with Ruby or other programming languages will probably want to use CASS as a library. Assuming you’ve installed CASS as a gem (see above), running a basic analysis with CASS is straightforward. First, we require the gem:

require ‘cass’ We don’t want the inconvenience of having to call all the methods through the Cass module (e.g., Cass::Contrast.new, Cass::Document.new, etc.), so let’s go ahead and include the contents of the module in the namespace:

include Cass

Now we can start running analyses. Let’s say we have a text file containing transcribed conversations of people discussing foods they like and dislike (e.g., cake.txt in the sample analysis package). Suppose we’re particularly interested in two foods: cake and spinach. Our goal is to test the hypothesis that people prefer cake to spinach. Operationally, we’re going to do that by examining the relative distance from ‘spinach’ and ‘cake’ to the terms ‘good’ and ‘bad’ in semantic space.

The first thing to do is set up the right contrasts. In this case, we’ll create a single contrast comparing the distance between cake and spinach with respect to good and bad:

contrast = Contrast.new(“cake spinach good bad”)

CASS interprets a string of four words as two ordered pairs: ‘cake’ and ‘spinach’ form one pair, ‘good’ and ‘bad’ the other (we could, equivalently, initialize the contrast by passing the 4-element array [‘cake’, ‘spinach’, ‘good’, ‘bad’]).

Next, we read the file containing the transcripts:

text = File.new(“cake.txt”).read And then we can create a corresponding Document. We initialize the Document object by passing a descriptive name, the contrasts we want to run, and the full text we want to analyze:

doc = Document.new(“cake_vs_spinach”, contrast, text)

If we want to see some information about the contents of our document, we can type:

doc.summary And that prints something like this to our screen:

> Summary for document ‘cake.txt’: > 4 target words (cake, spinach, good, bad) > 35 words in context. > Using 21 lines (containing at least one target word) for analysis.

Nothing too fancy, just basic descriptive information. The summary method has some additional arguments we could use to get more detailed information (e.g., word_count, list_context, etc.), but we’ll skip those for now.

Now if we want to compute the interaction term for our contrast (i.e., the difference of differences, reflecting the equation (cake.good - spinach.bad) - (cake.bad - spinach.good)), all we have to do is:

contrast.apply(doc)

And we get back something that looks like this:

0.5117 0.4039 0.3256 0.4511 0.2333

Where the first four values represent the similarity between the 4 pairs of words used to generate the interaction term (e.g., the first value reflects the correlation between ‘cake’ and ‘good’, the second between ‘spinach’ and ‘bad’, and so on), and the fifth is the interaction term. So in this case, the result (0.23) tells us that there’s a positive bias in the text, such that cake is semantically more closely related to good (relative to bad) than spinach is. Hypothesis confirmed!

Well, sort of. By itself, the number 0.23 doesn’t mean very much. We don’t know what the standard error is, so we have no idea whether 0.23 is a very large number or a very small one that might occur pretty often just by chance. Fortunately, we can generate some bootstrapped p values quite easily. First, we generate the bootstraps:

Analysis.bootstrap_test(doc, contrasts, “speech_results.txt”, 1000)

Here we call the bootstrap_test method, feeding it the document we want to analyze, the Contrasts we want to apply, the filename root we want to use, and the number of iterations we want to run (generally, as many as is computationally viable). The results will be saved to a plaintext file with the specified name, and we can peruse that file at our leisure. If we open it up, the first few lines look like this:

cake.spinach.good.bad observed cake.txt 0.5117 0.4039 0.3256 0.4511 0.2333 cake.spinach.good.bad boot_1 cake.txt 0.4118 0.2569 0.1481 0.4086 0.4154 cake.spinach.good.bad boot_2 cake.txt 0.5321 0.4353 0.4349 0.5396 0.2015 cake.spinach.good.bad boot_3 cake.txt 0.51 0.4216 0.3043 0.6923 0.4764 cake.spinach.good.bad boot_4 cake.txt 0.6222 0.3452 0.238 0.249 0.288 …

The columns tell us, respectively, what file the results came from, the bootstrap iteration (the first line shows us the actual, or observed value), and the observed interaction terms. Given this information, we can now compare the bootstrapped distribution to zero to test our hypothesis. We do that like this:

Analysis.p_values(“speech_results.txt”, ‘boot’)

…where the first argument specifies the full path to the file containing the bootstrap results we want to summarize, and the second argument indicates the type of test that was conducted (either ‘boot’ or ‘perm’). The results will be written to a file named speech_results_boot_p_values.txt. If we open that document up, we see this:

file contrast N value p-value cake.txt cake.spinach.good.bad 1000 0.2333 0.0 cake.txt mean 1000 0.2333 0.0

As you can see, the last column (p-value) reads 0.0, which is to say, none of the 1,000 iterations we ran had a value greater than 0. So we can reject the null hypothesis of zero effect at p < .001 in this case. Put differently, it’s exceedingly unlikely that we would get this result (people having a positive bias towards cake relative to spinach) just by chance. Of course, that’s a contrived example that won’t surprise anyone. But the point is that you can use the CASS tools in a similar way to ask other much more interesting questions about the relation between different terms in semantic space. So that’s the end of this overview; to learn more about the other functionality in CASS, read the [tutorial] document, or just surf around this RDoc.