Edwin Chen's Bloghttp://blog.echen.me/2022-02-11T00:00:00+01:00A Visual Tool for Exploring Word Embeddings2022-02-11T00:00:00+01:002022-02-11T00:00:00+01:00Edwin Chentag:blog.echen.me,2022-02-11:/2022/02/11/a-visual-tool-for-exploring-word-embeddings/<p>I built a visualization to explore embeddings a few years ago, but never posted it more broadly. So here it is! <a href="http://blog.echen.me/embedding-explorer/">http://blog.echen.me/embedding-explorer/</a></p>
<p>These are <a href="https://nlp.stanford.edu/projects/glove/">GloVe embeddings</a> projected into 2D, colorized via k-means in the original space.</p>
<p>You can see, for example, that the cluster in pink …</p><p>I built a visualization to explore embeddings a few years ago, but never posted it more broadly. So here it is! <a href="http://blog.echen.me/embedding-explorer/">http://blog.echen.me/embedding-explorer/</a></p>
<p>These are <a href="https://nlp.stanford.edu/projects/glove/">GloVe embeddings</a> projected into 2D, colorized via k-means in the original space.</p>
<p>You can see, for example, that the cluster in pink at the bottom right is a cluster of names.</p>
<p><img alt="Names Cluster" src="https://i.imgur.com/bEItyva.png"></p>
<p>And the cluster in red on the right is a cluster of geographical terms.</p>
<p><img alt="Geography Cluster" src="https://i.imgur.com/NtFaopz.png"></p>
<p>We can click on one of these points ("iceland") to see its nearest neighbors in the high-dimensional space (mostly other countries!) as well as other points that belong to the same cluster (Cluster 18 is this red cluster).</p>
<p><img alt="Iceland" src="https://i.imgur.com/4fggBQz.png"></p>
<p>We can also inspect each individual embedding dimension, to understand what it's picking up. Embedding Dimension 1, for example, seems to capture sportiness.</p>
<p><img alt="Embedding Dimension 1" src="https://i.imgur.com/YRhHkka.png"></p>
<p>We can also slide through the points based on their embedding dimension to get a better sense:</p>
<p><img alt="Embedding Dimension 1 Slide" src="https://i.imgur.com/Qq0ua2w.png"></p>
<p>Anyways, play around with the explorer <a href="http://blog.echen.me/embedding-explorer">here</a>, and feedback is always welcome!</p>Surge AI: A New Data Labeling Platform and Workforce for NLP2020-11-30T00:00:00+01:002020-11-30T00:00:00+01:00Edwin Chentag:blog.echen.me,2020-11-30:/2020/11/30/surge-ai-a-new-data-labeling-platform-and-workforce-for-nlp/<p><em><strong>tl;dr</strong> I started <a href="https://www.surgehq.ai">Surge AI</a> to fix the problems I've always encountered with getting <em>high-quality, human-labeled data at scale</em>. Think MTurk 2.0—but with an obsessive focus on quality and speed, and an elite workforce you can trust. If you've ever had problems getting human-annotated data, or wish …</em></p><p><em><strong>tl;dr</strong> I started <a href="https://www.surgehq.ai">Surge AI</a> to fix the problems I've always encountered with getting <em>high-quality, human-labeled data at scale</em>. Think MTurk 2.0—but with an obsessive focus on quality and speed, and an elite workforce you can trust. If you've ever had problems getting human-annotated data, or wish you had a labeling platform you could use with your own workforce, reach out at <a href="mailto:edwin@surgehq.ai">edwin@surgehq.ai</a> or get started <a href="https:/www.surgehq.ai">here</a>! We work with amazing AI companies like OpenAI, Airbnb, Amazon, and more on cutting-edge data and ML problems, and we'd love to help you as well.</em></p>
<p>Getting <strong>trustworthy human-labeled data</strong> has constantly been one of my biggest blockers, through a decade of working on real-world AI. Whether at Google, Facebook, or Twitter:</p>
<ul>
<li>Getting ground truth for training our ML models, and measuring their relevance and precision, invariably took months from backlogged teams of in-house labelers.</li>
<li>Speed and scale weren't the only issues—in-house labelers often weren't high quality either! When your labeling software is just an Excel spreadsheet, it's difficult to monitor and motivate performance.</li>
<li>The quality and speed of external labeling companies were even worse. We'd wait 3 months to get 10K rows of text labeled for a new NLP model, only to find that 50% of the labels were complete spam.</li>
</ul>
<p>This was a huge bottleneck for our AI and data science teams. If it takes months to get the data your ML needs, how can you iterate and move fast?</p>
<p>So I built our human evaluation platforms at YouTube and Twitter to fix these problems. And we used them to do amazing things: from <a href="https://blog.echen.me/2013/01/08/improving-twitter-search-with-real-time-human-computation">real-time human/AI search and advertising systems</a>, to <a href="https://blog.echen.me/2014/10/07/moving-beyond-ctr-better-recommendations-through-human-evaluation/">shifting our recommender objectives from clickbait to human preferences</a> (<a href="https://surge-ai.medium.com/we-need-new-infrastructure-for-the-next-ai-revolution-e1879f72550f">especially important in this age of political polarization!</a>), and more.</p>
<p>A cornerstone of these data labeling platforms was the premise that in order to build increasingly sophisticated real-world AI—for complex problems like hate speech and misinformation—we need skilled, motivated human workforces to measure and train them. So when COVID hit, and a huge educated population became out of a job or stuck at home, we realized there was an opportunity to turn the platforms we'd built into a startup, and created <a href="https://www.surgehq.ai">Surge AI</a>.</p>
<p>We've grown 10x in the past 6 months, and have worked with an amazing set of early customers on amazing things:</p>
<ul>
<li>We've helped companies <strong>boost their ML model performance by 50%</strong>, simply by relabeling their existing datasets.</li>
<li>Our customers have dropped their time waiting for new labels—their biggest bottleneck in training new models—from <strong>3-6 months to just a few days</strong>.</li>
<li>We label millions of images and pieces of text every week, in over a <strong>dozen languages</strong>.</li>
<li>We're not limited to content labeling: we help our customers with everything from <strong>content moderation and AI fairness, to making phone calls to gather business and medical information, to core image recognition and NLP</strong>.</li>
<li>Teams even use our API to gather human judgments in real time!</li>
</ul>
<p>So if you need a high-quality labeling workforce you can trust, or want to use our labeling software for your own agents, please <a href="mailto:edwin@surgehq.ai">reach out</a> or <a href="https:/www.surgehq.ai">get started</a>! I'd love to help you get the datasets and machine learning models you need, or even just hear more about your own problems and experiences.</p>How Could Facebook Align its ML Systems to Human Values? A Data-Driven Approach2020-02-12T00:00:00+01:002020-02-12T00:00:00+01:00Edwin Chentag:blog.echen.me,2020-02-12:/2020/02/12/how-could-facebook-align-its-ml-systems-to-human-values-a-data-driven-approach/<p><em>This is a <a href="https://www.surgehq.ai/blog/what-if-social-media-optimized-for-human-values">crosspost</a> from the official Surge AI <a href="https://www.surgehq.ai/blog/what-if-social-media-optimized-for-human-values">blog</a>.</em></p>
<p>For background on this blog post...</p>
<p>I used to work at Facebook, YouTube, and Twitter. One of the problems I big worked on: what was the right objective function to align our AI systems towards?</p>
<p>Optimizing for watch time at …</p><p><em>This is a <a href="https://www.surgehq.ai/blog/what-if-social-media-optimized-for-human-values">crosspost</a> from the official Surge AI <a href="https://www.surgehq.ai/blog/what-if-social-media-optimized-for-human-values">blog</a>.</em></p>
<p>For background on this blog post...</p>
<p>I used to work at Facebook, YouTube, and Twitter. One of the problems I big worked on: what was the right objective function to align our AI systems towards?</p>
<p>Optimizing for watch time at YouTube, for example, led to longer videos for the sake of longer videos, and clicks on videos with racy thumbnails.</p>
<p>Optimizing engagement at Facebook led to low-quality clickbait, and Hooter's appearing as the top search result when you searched for restaurants in Houston.</p>
<p>Optimizing for favorites and replies at Twitter led to toxic content at the top of the feed.</p>
<p>So while watch time, engagement, and replies would always go up – were these really the products we wanted to build? What happened to Facebook's original mission of connecting users with their friends and family? What did "favorites" have to do with being <em>the</em> platform for public conversation at Twitter? A ton of engineering and data science time is spent measuring active users, but why were there no dashboards measuring progress to broader product goals?</p>
<p>So could we figure out a metric that was better tuned to human values and preferences – but also fast, rigorous, and easily measurable? After all, we still need our A/B tests, ML objectives, and OKRs to align the company around.</p>
<p>This question is particularly important today, with all the troubles that social media platforms face, so I wrote up an approach I've often worked on, based on Facebook data. Read it <a href="https://www.surgehq.ai/blog/what-if-social-media-optimized-for-human-values">on the Surge AI blog</a>!</p>Exploring LSTMs2017-05-30T00:00:00+02:002017-05-30T00:00:00+02:00Edwin Chentag:blog.echen.me,2017-05-30:/2017/05/30/exploring-lstms/<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/gist-embed/2.4/gist-embed.min.js"></script>
<p>LSTMs are behind a lot of the amazing achievements deep learning has made in the past few years, and they're a fairly simple extension to neural networks under the right view. So I'll try to present them as intuitively as possible – in such a way that you could have discovered …</p><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/gist-embed/2.4/gist-embed.min.js"></script>
<p>LSTMs are behind a lot of the amazing achievements deep learning has made in the past few years, and they're a fairly simple extension to neural networks under the right view. So I'll try to present them as intuitively as possible – in such a way that you could have discovered them yourself.</p>
<p>But first, a picture:</p>
<p><img alt="LSTM" src="http://i.imgur.com/lnBi8OZ.png"></p>
<p>Aren't LSTMs beautiful? Let's go.</p>
<p>(Note: if you're already familiar with neural networks and LSTMs, skip to the middle – the first half of this post is a tutorial.)</p>
<h1>Neural Networks</h1>
<p>Imagine we have a sequence of images from a movie, and we want to label each image with an activity (is this a fight?, are the characters talking?, are the characters eating?).</p>
<p>How do we do this?</p>
<p>One way is to ignore the sequential nature of the images, and build a <em>per-image</em> classifier that considers each image in isolation. For example, given enough images and labels:</p>
<ul>
<li>Our algorithm might first learn to detect low-level patterns like shapes and edges. </li>
<li>With more data, it might learn to combine these patterns into more complex ones, like faces (two circular things atop a triangular thing atop an oval thing) or cats.</li>
<li>And with even more data, it might learn to map these higher-level patterns into activities themselves (scenes with mouths, steaks, and forks are probably about eating).</li>
</ul>
<p>This, then, is a <strong>deep neural network</strong>: it takes an image input, returns an activity output, and – just as we might learn to detect patterns in puppy behavior without knowing anything about dogs (after seeing enough corgis, we discover common characteristics like fluffy butts and drumstick legs; next, we learn advanced features like splooting) – in between it learns to represent images through hidden layers of representations.</p>
<h2>Mathematically</h2>
<p>I assume people are familiar with basic neural networks already, but let's quickly review them.</p>
<ul>
<li>A <strong>neural network</strong> with a single hidden layer takes as input a vector x, which we can think of as a set of neurons. </li>
<li>Each input neuron is connected to a hidden layer of neurons via a set of learned weights. </li>
<li>The jth hidden neuron outputs <span class="math">\(h_j = \phi(\sum_i w_{ij} x_i)\)</span>, where <span class="math">\(\phi\)</span> is an activation function. </li>
<li>The hidden layer is fully connected to an output layer, and the jth output neuron outputs <span class="math">\(y_j = \sum_i v_{ij} h_i\)</span>. If we need probabilities, we can transform the output layer via a <a href="https://en.wikipedia.org/wiki/Softmax_function">softmax</a> function.</li>
</ul>
<p>In matrix notation:</p>
<div class="math">$$h = \phi(Wx)$$</div>
<div class="math">$$y = Vh$$</div>
<p>where</p>
<ul>
<li>x is our input vector</li>
<li>W is a weight matrix connecting the input and hidden layers</li>
<li>V is a weight matrix connecting the hidden and output layers</li>
<li>Common activation functions for <span class="math">\(\phi\)</span> are the <a href="https://en.wikipedia.org/wiki/Sigmoid_function">sigmoid function</a>, <span class="math">\(\sigma(x)\)</span>, which squashes numbers into the range (0, 1); the <a href="https://en.wikipedia.org/wiki/Hyperbolic_function#Standard_analytic_expressions">hyperbolic tangent</a>, <span class="math">\(tanh(x)\)</span>, which squashes numbers into the range (-1, 1), and the <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)">rectified linear unit</a>, <span class="math">\(ReLU(x) = max(0, x)\)</span>.</li>
</ul>
<p>Here's a pictorial view:</p>
<p><img alt="Neural Network" src="http://i.imgur.com/tFMQMOn.png"></p>
<p>(Note: to make the notation a little cleaner, I assume x and h each contain an extra bias neuron fixed at 1 for learning bias weights.)</p>
<h1>Remembering Information with RNNs</h1>
<p>Ignoring the sequential aspect of the movie images is pretty ML 101, though. If we see a scene of a beach, we should boost beach activities in future frames: an image of someone in the water should probably be labeled <em>swimming</em>, not <em>bathing</em>, and an image of someone lying with their eyes closed is probably <em>suntanning</em>. If we remember that Bob just arrived at a supermarket, then even without any distinctive supermarket features, an image of Bob holding a slab of bacon should probably be categorized as <em>shopping</em> instead of <em>cooking</em>.</p>
<p>So what we'd like is to let our model track the state of the world:</p>
<ol>
<li><strong>After seeing each image, the model outputs a label and also updates the knowledge it's been learning.</strong> For example, the model might learn to automatically discover and track information like location (are scenes currently in a house or beach?), time of day (if a scene contains an image of the moon, the model should remember that it's nighttime), and within-movie progress (is this image the first frame or the 100th?). Importantly, just as a neural network automatically discovers hidden patterns like edges, shapes, and faces without being fed them, our model should automatically discover useful information by itself.</li>
<li>When given a new image, the model should <strong>incorporate the knowledge it's gathered</strong> to do a better job.</li>
</ol>
<p>This, then, is a <strong>recurrent neural network</strong>. Instead of simply taking an image and returning an activity, an RNN also maintains internal memories about the world (weights assigned to different pieces of information) to help perform its classifications.</p>
<h2>Mathematically</h2>
<p>So let's add the notion of <strong>internal knowledge</strong> to our equations, which we can think of as pieces of information that the network maintains over time. </p>
<p>But this is easy: we know that the hidden layers of neural networks already encode useful information about their inputs, so why not use these layers as the memory passed from one time step to the next? This gives us our RNN equations:</p>
<div class="math">$$h_t = \phi(Wx_t + Uh_{t-1})$$</div>
<div class="math">$$y_t = Vh_t$$</div>
<p>Note that the hidden state computed at time <span class="math">\(t\)</span> (<span class="math">\(h_t\)</span>, our internal knowledge) is fed back at the next time step. (Also, I'll use concepts like hidden state, knowledge, memories, and beliefs to describe <span class="math">\(h_t\)</span> interchangeably.)</p>
<p><img alt="RNN" src="http://i.imgur.com/ifQrKRR.png"></p>
<h1>Longer Memories through LSTMs</h1>
<p>Let's think about how our model updates its knowledge of the world. So far, we've placed no constraints on this update, so its knowledge can change pretty chaotically: at one frame it thinks the characters are in the US, at the next frame it sees the characters eating sushi and thinks they're in Japan, and at the next frame it sees polar bears and thinks they're on Hydra Island. Or perhaps it has a wealth of information to suggest that Alice is an investment analyst, but decides she's a professional assassin after seeing her cook.</p>
<p>This chaos means information quickly transforms and vanishes, and it's difficult for the model to keep a long-term memory. So what we'd like is for the network to <em>learn</em> how to update its beliefs (scenes without Bob shouldn't change Bob-related information, scenes with Alice should focus on gathering details about her), in a way that its knowledge of the world evolves more gently. </p>
<p>This is how we do it.</p>
<ol>
<li><strong>Adding a forgetting mechanism.</strong> If a scene ends, for example, the model should forget the current scene location, the time of day, and reset any scene-specific information; however, if a character dies in the scene, it should continue remembering that he's no longer alive. Thus, we want the model to learn a separate <em>forgetting/remembering</em> mechanism: when new inputs come in, it needs to know which beliefs to keep or throw away.</li>
<li><strong>Adding a saving mechanism.</strong> When the model sees a new image, it needs to learn whether any information about the image is worth using and saving. Maybe your mom sent you an article about the Kardashians, but who cares?</li>
<li>So when new a input comes in, the model first forgets any long-term information it decides it no longer needs. Then it learns which parts of the new input are worth using, and saves them into its long-term memory.</li>
<li><strong>Focusing long-term memory into working memory.</strong> Finally, the model needs to learn which parts of its long-term memory are immediately useful. For example, Bob's age may be a useful piece of information to keep in the long term (children are more likely to be crawling, adults are more likely to be working), but is probably irrelevant if he's not in the current scene. So instead of using the full long-term memory all the time, it learns which parts to focus on instead.</li>
</ol>
<p>This, then, is an <strong>long short-term memory network</strong>. Whereas an RNN can overwrite its memory at each time step in a fairly uncontrolled fashion, an LSTM transforms its memory in a very precise way: by using <em>specific learning mechanisms</em> for which pieces of information to remember, which to update, and which to pay attention to. This helps it keep track of information over longer periods of time.</p>
<h2>Mathematically</h2>
<p>Let's describe the LSTM additions mathematically.</p>
<p>At time <span class="math">\(t\)</span>, we receive a new input <span class="math">\(x_t\)</span>. We also have our long-term and working memories passed on from the previous time step, <span class="math">\(ltm_{t-1}\)</span> and <span class="math">\(wm_{t-1}\)</span> (both n-length vectors), which we want to update.</p>
<p><strong>We'll start with our long-term memory.</strong> First, we need to know which pieces of long-term memory to continue remembering and which to discard, so we want to use the new input and our working memory to learn a <strong>remember gate</strong> of n numbers between 0 and 1, each of which determines how much of a long-term memory element to keep. (A 1 means to keep it, a 0 means to forget it entirely.)</p>
<p>Naturally, we can use a small neural network to learn this remember gate:</p>
<div class="math">$$remember_t = \sigma(W_r x_t + U_r wm_{t-1}) $$</div>
<p>(Notice the similarity to our previous network equations; this is just a shallow neural network. Also, we use a sigmoid activation because we need numbers between 0 and 1.)</p>
<p>Next, we need to compute the information we can learn from <span class="math">\(x_t\)</span>, i.e., a <strong>candidate addition to our long-term memory</strong>:</p>
<div class="math">$$ ltm'_t = \phi(W_l x_t + U_l wm_{t-1}) $$</div>
<p><span class="math">\(\phi\)</span> is an activation function, commonly chosen to be <span class="math">\(tanh\)</span>.</p>
<p>Before we add the candidate into our memory, though, we want to learn <strong>which parts of it are actually worth using and saving</strong>:</p>
<div class="math">$$save_t = \sigma(W_s x_t + U_s wm_{t-1})$$</div>
<p>(Think of what happens when you read something on the web. While a news article might contain information about Hillary, you should ignore it if the source is Breitbart.)</p>
<p>Let's now combine all these steps. After forgetting memories we don't think we'll ever need again and saving useful pieces of incoming information, we have our <strong>updated long-term memory</strong>:</p>
<div class="math">$$ltm_t = remember_t \circ ltm_{t-1} + save_t \circ ltm'_t$$</div>
<p>where <span class="math">\(\circ\)</span> denotes element-wise multiplication.</p>
<p><strong>Next, let's update our working memory.</strong> We want to learn how to focus our long-term memory into information that will be <em>immediately</em> useful. (Put differently, we want to learn what to move from an <em>external hard drive</em> onto our <em>working laptop</em>.) So we learn a <strong>focus/attention vector</strong>:</p>
<div class="math">$$focus_t = \sigma(W_f x_t + U_f wm_{t-1})$$</div>
<p>Our <strong>working memory</strong> is then</p>
<div class="math">$$wm_t = focus_t \circ \phi(ltm_t)$$</div>
<p>In other words, we pay full attention to elements where the focus is 1, and ignore elements where the focus is 0.</p>
<p>And we're done! Hopefully this made it into your long-term memory as well.</p>
<hr>
<p>To summarize, whereas a vanilla RNN uses one equation to update its hidden state/memory:</p>
<div class="math">$$h_t = \phi(Wx_t + Uh_{t-1})$$</div>
<p>An LSTM uses several:</p>
<div class="math">$$ltm_t = remember_t \circ ltm_{t-1} + save_t \circ ltm'_t$$</div>
<div class="math">$$wm_t = focus_t \circ tanh(ltm_t)$$</div>
<p>where each memory/attention sub-mechanism is just a mini brain of its own:</p>
<div class="math">$$remember_t = \sigma(W_r x_t+ U_r wm_{t-1}) $$</div>
<div class="math">$$save_t = \sigma(W_s x_t + U_s wm_{t-1})$$</div>
<div class="math">$$focus_t = \sigma(W_f x_t + U_f wm_{t-1})$$</div>
<div class="math">$$ ltm'_t = tanh(W_l x_t + U_l wm_{t-1}) $$</div>
<p>(Note: the terminology and variable names I've been using are different from the usual literature. Here are the standard names, which I'll use interchangeably from now on:</p>
<ul>
<li>The long-term memory, <span class="math">\(ltm_t\)</span>, is usually called the <strong>cell state</strong>, denoted <span class="math">\(c_t\)</span>.</li>
<li>The working memory, <span class="math">\(wm_t\)</span>, is usually called the <strong>hidden state</strong>, denoted <span class="math">\(h_t\)</span>. This is analogous to the hidden state in vanilla RNNs.</li>
<li>The remember vector, <span class="math">\(remember_t\)</span>, is usually called the <strong>forget gate</strong> (despite the fact that a 1 in the forget gate still means to keep the memory and a 0 still means to forget it), denoted <span class="math">\(f_t\)</span>.</li>
<li>The save vector, <span class="math">\(save_t\)</span>, is usually called the <strong>input gate</strong> (as it determines how much of the input to let into the cell state), denoted <span class="math">\(i_t\)</span>.</li>
<li>The focus vector, <span class="math">\(focus_t\)</span>, is usually called the <strong>output gate</strong>, denoted <span class="math">\(o_t\)</span>.
)</li>
</ul>
<p><img alt="LSTM" src="http://i.imgur.com/vsqgLYn.png"></p>
<h1>Snorlax</h1>
<p>I could have caught a hundred Pidgeys in the time it took me to write this post, so here's a cartoon.</p>
<h2>Neural Networks</h2>
<p><img alt="Neural Network" src="http://i.imgur.com/cOGzJxk.png"></p>
<h2>Recurrent Neural Networks</h2>
<p><img alt="RNN" src="http://i.imgur.com/PnWiSCf.png"></p>
<h2>LSTMs</h2>
<p><img alt="LSTM" src="http://i.imgur.com/EGZIUuc.pngg"></p>
<h2>Learning to Code</h2>
<p>Let's look at a few examples of what an LSTM can do. Following Andrej Karpathy's <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">terrific post</a>, I'll use character-level LSTM models that are fed sequences of characters and trained to predict the next character in the sequence. </p>
<p>While this may seem a bit toyish, character-level models can actually be very useful, even on top of word models. For example:</p>
<ul>
<li><strong>Imagine a code autocompleter smart enough to allow you to program on your phone.</strong> An LSTM could (in theory) track the return type of the method you're currently in, and better suggest which variable to return; it could also know without compiling whether you've made a bug by returning the wrong type.</li>
<li><strong>NLP applications like machine translation often have trouble dealing with rare terms.</strong> How do you translate a word you've never seen before, or convert adjectives to adverbs? Even if you know what a tweet means, how do you generate a new hashtag to capture it? Character models can daydream new terms, so this is another area with <a href="https://arxiv.org/pdf/1508.07909.pdf">interesting applications</a>.</li>
</ul>
<p>So to start, I spun up an EC2 p2.xlarge spot instance, and trained a 3-layer LSTM on the <a href="https://github.com/apache/commons-lang">Apache Commons Lang codebase</a>. Here's a program it generates after a few hours.</p>
<p><code data-gist-id="de1d2d1bd34a94fba100b57b3a9e49ab" data-gist-hide-footer="true"></code></p>
<p>While the code certainly isn't perfect, it's better than a lot of data scientists I know. And we can see that the LSTM has learned a lot of interesting (and correct!) coding behavior:</p>
<ul>
<li><strong>It knows how to structure classes:</strong> a license up top, followed by packages and imports, followed by comments and a class definition, followed by variables and methods. Similarly, it knows how to create methods: comments follow the correct orders (description, then @param, then @return, etc.), decorators are properly placed, and non-void methods end with appropriate return statements. Crucially, this behavior spans long ranges of code – see how giant the blocks are!</li>
<li><strong>It can also track subroutines and nesting levels:</strong> indentation is always correct, and if statements and for loops are always closed out.</li>
<li><strong>It even knows how to create tests.</strong></li>
</ul>
<p>How does the model do this? Let's look at a few of the hidden states.</p>
<p>Here's a neuron that seems to track the code's <em>outer</em> level of indentation:</p>
<p>(As the LSTM moves through the sequence, its neurons fire at varying intensities. The picture represents one particular neuron, where each row is a sequence and characters are color-coded according to the neuron's intensity; dark blue shades indicate large, positive activations, and dark red shades indicate very negative activations.)</p>
<p><img alt="Outer Level of Indentation" src="http://i.imgur.com/szJnfAY.png"></p>
<p>And here's a neuron that counts down the spaces between tabs:</p>
<p><img alt="Tab Spaces" src="http://i.imgur.com/EBbM9Kx.png"></p>
<p>For kicks, here's the output of a different 3-layer LSTM trained on TensorFlow's codebase:</p>
<p><code data-gist-id="926b34e5bec6cd2b304cdecabda255b1" data-gist-hide-footer="true"></code></p>
<p>There are <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">plenty of other fun examples</a> floating around the web, so check them out if you want to see more.</p>
<h1>Investigating LSTM Internals</h1>
<p>Let's dig a little deeper. We looked in the last section at examples of hidden states, but I wanted to play with LSTM cell states and their other memory mechanisms too. Do they fire when we expect, or are there surprising patterns?</p>
<h2>Counting</h2>
<p>To investigate, let's start by teaching an LSTM to count. (Remember how the Java and Python LSTMs were able to generate proper indentation!) So I generated sequences of the form</p>
<div class="highlight"><pre><span></span><code><span class="err">aaaaaXbbbbb</span>
</code></pre></div>
<p>(N "a" characters, followed by a delimiter X, followed by N "b" characters, where 1 <= N <= 10), and trained a single-layer LSTM with 10 hidden neurons.</p>
<p>As expected, the LSTM learns perfectly within its training range – and can even generalize a few steps beyond it. (Although it starts to fail once we try to get it to count to 19.)</p>
<div class="highlight"><pre><span></span><code><span class="err">aaaaaaaaaaaaaaaXbbbbbbbbbbbbbbb</span>
<span class="err">aaaaaaaaaaaaaaaaXbbbbbbbbbbbbbbbb</span>
<span class="err">aaaaaaaaaaaaaaaaaXbbbbbbbbbbbbbbbbb</span>
<span class="err">aaaaaaaaaaaaaaaaaaXbbbbbbbbbbbbbbbbbb</span>
<span class="err">aaaaaaaaaaaaaaaaaaaXbbbbbbbbbbbbbbbbbb # Here it begins to fail: the model is given 19 "a"s, but outputs only 18 "b"s.</span>
</code></pre></div>
<p>We expect to find a hidden state neuron that counts the number of a's if we look at its internals. And we do:</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=counter&layer=1&n=2"><img alt="Neuron #2 Hidden State" src="http://i.imgur.com/JEIhBeh.png"></a></p>
<p>I built a <a href="http://blog.echen.me/lstm-explorer">small web app</a> to play around with LSTMs, and <a href="http://blog.echen.me/lstm-explorer/#/neuron?file=counter&layer=1&n=2">Neuron #2</a> seems to be counting both the number of a's it's seen, as well as the number of b's. (Remember that cells are shaded according to the neuron's activation, from dark red [-1] to dark blue [+1].)</p>
<p>What about the cell state? It behaves similarly:</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=counter&layer=1&n=2"><img alt="Neuron #2 Cell State" src="http://i.imgur.com/5DB3wnJ.png"></a></p>
<p>One interesting thing is that the working memory looks like a "sharpened" version of the long-term memory. Does this hold true in general? </p>
<p>It does. (This is exactly as we would expect, since the long-term memory gets squashed by the tanh activation function and the output gate limits what gets passed on.) For example, here is an overview of all 10 cell state nodes at once. We see plenty of light-colored cells, representing values close to 0.</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/network?file=counter&layer=1"><img alt="Counting LSTM Cell States" src="http://i.imgur.com/yGfaCh3.png"></a></p>
<p>In contrast, <strong>the 10 working memory neurons look much more focused</strong>. Neurons 1, 3, 5, and 7 are even zeroed out entirely over the first half of the sequence.</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/network?file=counter&vector=hs&layer=1"><img alt="Counting LSTM Hidden States" src="http://i.imgur.com/scQXaPL.png"></a></p>
<p>Let's go back to Neuron #2. Here are the candidate memory and input gate. They're relatively constant over each half of the sequence – as if the neuron is calculating <code>a += 1</code> or <code>b += 1</code> at each step.</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=counter&layer=1&n=2"><img alt="Counting LSTM Candidate Memory" src="http://i.imgur.com/Nu504g0.png"></a></p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=counter&layer=1&n=2"><img alt="Input Gate" src="http://i.imgur.com/iBkuAiV.png"></a></p>
<p>Finally, here's an overview of all of Neuron 2's internals:</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=counter&layer=1&n=2"><img alt="Neuron 2 Overview" src="http://i.imgur.com/VUwMnHJ.png"></a></p>
<p>If you want to investigate the different counting neurons yourself, you can play around with the visualizer <a href="http://blog.echen.me/lstm-explorer/#/network?file=counter">here</a>.</p>
<iframe src="http://blog.echen.me/lstm-explorer/#/network?file=counter" style="width: 100%; height: 500px"></iframe>
<p>(Note: this is far from the only way an LSTM can learn to count, and I'm anthropomorphizing quite a bit here. But I think viewing the network's behavior is interesting and can help build better models – after all, many of the ideas in neural networks come from analogies to the human brain, and if we see unexpected behavior, we may be able to design more efficient learning mechanisms.)</p>
<h2>Count von Count</h2>
<p>Let's look at a slightly more complicated counter. This time, I generated sequences of the form</p>
<div class="highlight"><pre><span></span><code><span class="err">aaXaXaaYbbbbb</span>
</code></pre></div>
<p>(N a's with X's randomly sprinkled in, followed by a delimiter Y, followed by N b's). The LSTM still has to count the number of a's, but this time needs to ignore the X's as well. </p>
<p><a href="http://blog.echen.me/lstm-explorer/#/network?file=selective_counter">Here's the full LSTM.</a> We expect to see <strong>a counting neuron, but one where the input gate is zero whenever it sees an X.</strong> And we do!</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=selective_counter&layer=1&n=20"><img alt="Counter 2 - Cell State" src="http://i.imgur.com/fFs0MNm.png"></a></p>
<p>Above is the cell state of <a href="http://blog.echen.me/lstm-explorer/#/neuron?file=selective_counter&layer=1&n=20">Neuron 20</a>. It increases until it hits the delimiter Y, and then decreases to the end of the sequence – just like it's calculating a <code>num_bs_left_to_print</code> variable that increments on a's and decrements on b's.</p>
<p>If we look at its input gate, it is indeed ignoring the X's:</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=selective_counter&layer=1&n=20"><img alt="Counter 2 - Input Gate" src="http://i.imgur.com/ccqdcn8.png"></a></p>
<p>Interestingly, though, the candidate memory fully activates on the irrelevant X's – which shows why the input gate is needed. (Although, if the input gate weren't part of the architecture, presumably the network would have presumably learned to ignore the X's some other way, at least for this simple example.)</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=selective_counter&layer=1&n=20"><img alt="Counter 2 - Candidate Memory" src="http://i.imgur.com/wmjemE1.png"></a></p>
<p>Let's also look at <a href="http://blog.echen.me/lstm-explorer/#/neuron?file=selective_counter&layer=1&n=10">Neuron 10</a>.</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=selective_counter&layer=1&n=10"><img alt="Counter 2 - Neuron 10" src="http://i.imgur.com/02ySRxl.png"></a></p>
<p>This neuron is interesting as it only activates when reading the delimiter "Y" – and yet it still manages to encode the number of a's seen so far in the sequence. (It may be hard to tell from the picture, but when reading Y's belonging to sequences with the same number of a's, all the cell states have values either identical or within 0.1% of each other. You can see that Y's with fewer a's are lighter than those with more.) Perhaps some other neuron sees Neuron 10 slacking and helps a buddy out.</p>
<iframe src="http://blog.echen.me/lstm-explorer/#/neuron?file=selective_counter&layer=1&n=20" style="width: 100%; height: 500px"></iframe>
<h2>Remembering State</h2>
<p>Next, I wanted to look at how LSTMs remember state. I generated sequences of the form</p>
<div class="highlight"><pre><span></span><code><span class="err">AxxxxxxYa</span>
<span class="err">BxxxxxxYb</span>
</code></pre></div>
<p>(i.e., an "A" or B", followed by 1-10 x's, then a delimiter "Y", ending with a lowercase version of the initial character). This way <a href="http://blog.echen.me/lstm-explorer/#/network?file=state_memorizer">the network</a> needs to remember whether it's in an "A" or "B" state.</p>
<p>We expect to find a neuron that fires when remembering that the sequence started with an "A", and another neuron that fires when remembering that it started with a "B". We do.</p>
<p>For example, here is an "A" neuron that activates when it reads an "A", and remembers until it needs to generate the final character. Notice that the input gate ignores all the "x" characters in between.</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=state_memorizer&layer=1&n=8"><img alt="A Neuron - #8" src="http://i.imgur.com/UfGWJ9w.png"></a></p>
<p>Here is its "B" counterpart:</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=state_memorizer&layer=1&n=17"><img alt="B Neuron - #17" src="http://i.imgur.com/pAKw9Y2.png"></a></p>
<p>One interesting point is that even though knowledge of the A vs. B state isn't needed until the network reads the "Y" delimiter, the hidden state fires throughout all the intermediate inputs anyways. This seems a bit "inefficient", but perhaps it's because the neurons are doing a bit of double-duty in counting the number of x's as well.</p>
<iframe src="http://blog.echen.me/lstm-explorer/#/neuron?file=state_memorizer&layer=1&n=8" style="width: 100%; height: 500px"></iframe>
<h2>Copy Task</h2>
<p>Finally, let's look at how an LSTM learns to copy information. (Recall that our Java LSTM was able to memorize and copy an Apache license.)</p>
<p>(Note: if you think about how LSTMs work, remembering lots of individual, detailed pieces of information isn't something they're very good at. For example, you may have noticed that one major flaw of the LSTM-generated code was that it often made use of undefined variables – the LSTMs couldn't remember which variables were in scope. This isn't surprising, since it's hard to use single cells to efficiently encode multi-valued information like characters, and LSTMs don't have a natural mechanism to chain adjacent memories to form words. <a href="https://arxiv.org/abs/1503.08895">Memory networks</a> and <a href="https://arxiv.org/abs/1410.5401">neural Turing machines</a> are two extensions to neural networks that help fix this, by augmenting with external memory components. So while copying isn't something LSTMs do very efficiently, it's fun to see how they try anyways.)</p>
<p>For this copy task, I trained a tiny 2-layer LSTM on sequences of the form</p>
<div class="highlight"><pre><span></span><code><span class="err">baaXbaa</span>
<span class="err">abcXabc</span>
</code></pre></div>
<p>(i.e., a 3-character subsequence composed of a's, b's, and c's, followed by a delimiter "X", followed by the same subsequence).</p>
<p>I wasn't sure what "copy neurons" would look like, so in order to find neurons that were memorizing parts of the initial subsequence, I looked at their hidden states when reading the delimiter X. Since the network needs to encode the initial subsequence, its states should exhibit different patterns depending on what they're learning.</p>
<p>The graph below, for example, plots Neuron 5's hidden state when reading the "X" delimiter. The neuron is clearly able to distinguish sequences beginning with a "c" from those that don't.</p>
<p><img alt="Neuron 5" src="http://i.imgur.com/QdQXIVu.png"></p>
<p>For another example, here is Neuron 20's hidden state when reading the "X". It looks like it picks out sequences beginning with a "b".</p>
<p><img alt="Neuron 20 Hidden State" src="http://i.imgur.com/ud8NjEA.png"></p>
<p>Interestingly, if we look at Neuron 20's <em>cell</em> state, it almost seems to capture the entire 3-character subsequence by itself (no small feat given its one-dimensionality!):</p>
<p><img alt="Neuron 20 Cell State" src="http://i.imgur.com/PLpSocg.png"></p>
<p>Here are <a href="http://blog.echen.me/lstm-explorer/#/neuron?file=copy_machine&layer=1&n=20">Neuron 20's cell and hidden states</a>, across the entire sequence. Notice that <strong>its hidden state is turned off over the entire initial subsequence</strong> (perhaps expected, since its memory only needs to be passively kept at that point).</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=copy_machine&layer=1&n=20"><img alt="Copy LSTM - Neuron 20 Hidden and Cell" src="http://i.imgur.com/dO0Ai6H.png"></a></p>
<p>However, if we look more closely, the neuron actually seems to be firing whenever the <em>next</em> character is a "b". So rather than being a "the sequence started with a b" neuron, it appears to be a "the next character is a b" neuron.</p>
<p>As far as I can tell, this pattern holds across the network – all the neurons seem to be predicting the next character, rather than memorizing characters at specific positions. For example, <a href="http://blog.echen.me/lstm-explorer/#/neuron?file=copy_machine&layer=1&n=5">Neuron 5</a> seems to be a "next character is a c" predictor.</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=copy_machine&layer=1&n=5"><img alt="Copy LSTM - Neuron 5" src="http://i.imgur.com/30LKz6j.png"></a></p>
<p>I'm not sure if this is the default kind of behavior LSTMs learn when copying information, or what other copying mechanisms are available as well.</p>
<iframe src="http://blog.echen.me/lstm-explorer/#/network?file=copy_machine" style="width: 100%; height: 500px"></iframe>
<h1>States and Gates</h1>
<p>To really hone in and understand the purpose of the different states and gates in an LSTM, let's repeat the previous section with a small pivot.</p>
<h2>Cell State and Hidden State (Memories)</h2>
<p>We originally described the cell state as a long-term memory, and the hidden state as a way to pull out and focus these memories when needed.</p>
<p>So when a memory is currently irrelevant, we expect the hidden state to turn off – and that's exactly what happens for this sequence copying neuron.</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=copy_machine&layer=1&n=5"><img alt="Copy Machine" src="http://i.imgur.com/30LKz6j.png"></a></p>
<h2>Forget Gate</h2>
<p>The forget gate discards information from the cell state (0 means to completely forget, 1 means to completely remember), so we expect it to fully activate when it needs to remember something exactly, and to turn off when information is never going to be needed again.</p>
<p>That's what we see with this "A" memorizing neuron: the forget gate fires hard to remember that it's in an "A" state while it passes through the x's, and turns off once it's ready to generate the final "a".</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=state_memorizer&layer=1&n=8"><img alt="Forget Gate" src="http://i.imgur.com/vAMXZdN.png"></a></p>
<h2>Input Gate (Save Gate)</h2>
<p>We described the job of the input gate (what I originally called the save gate) as deciding whether or not to save information from a new input. Thus, it should turn off at useless information.</p>
<p>And that's what this selective counting neuron does: it counts the a's and b's, but ignores the irrelevant x's.</p>
<p><a href="http://blog.echen.me/lstm-explorer/#/neuron?file=selective_counter&layer=1&n=20"><img alt="Input Gate" src="http://i.imgur.com/ccqdcn8.png"></a></p>
<p>What's amazing is that nowhere in our LSTM equations did we specify that this is how the input (save), forget (remember), and output (focus) gates should work. The network just learned what's best.</p>
<h1>Extensions</h1>
<p>Now let's recap how you could have discovered LSTMs by yourself.</p>
<p>First, many of the problems we'd like to solve are sequential or temporal of some sort, so we should incorporate past learnings into our models. But we already know that the hidden layers of neural networks encode useful information, so why not use these hidden layers as the memories we pass from one time step to the next? <strong>And so we get RNNs.</strong></p>
<p>But we know from our own behavior that we don't keep track of knowledge willy-nilly; when we read a new article about politics, we don't immediately believe whatever it tells us and incorporate it into our beliefs of the world. We selectively decide what information to save, what information to discard, and what pieces of information to use to make decisions the next time we read the news. Thus, we want to <em>learn</em> how to gather, update, and apply information – and why not learn these things through their own mini neural networks? <strong>And so we get LSTMs.</strong></p>
<p>And now that we've gone through this process, we can come up with our own modifications.</p>
<ul>
<li>For example, maybe you think it's silly for LSTMs to distinguish between long-term and working memories – why not have one? Or maybe you find separate remember gates and save gates kind of redundant – anything we forget should be replaced by new information, and vice-versa. <strong>And now you've come up with one popular LSTM variant, the <a href="https://arxiv.org/abs/1412.3555">GRU</a></strong>.</li>
<li>Or maybe you think that when deciding what information to remember, save, and focus on, we shouldn't rely on our working memory alone – why not use our long-term memory as well? <strong>And now you've discovered <a href="http://machinelearning.wustl.edu/mlpapers/paper_files/GersSS02.pdf">Peephole LSTMs</a></strong>.</li>
</ul>
<h1>Making Neural Nets Great Again</h1>
<p>Let's look at one final example, using a 2-layer LSTM trained on Trump's tweets. Despite the <del>tiny</del> <em>big</em> dataset, it's enough to learn a lot of patterns.</p>
<p>For example, here's a neuron that tracks its position within hashtags, URLs, and @mentions:</p>
<p><a href="http://i.imgur.com/eBJQLOf.png"><img alt="Hashtags, URLs, @mentions" src="http://i.imgur.com/eBJQLOf.png"></a></p>
<p>Here's a proper noun detector (note that it's not simply firing at capitalized words):</p>
<p><a href="http://i.imgur.com/SC4XMHr.png"><img alt="Proper Nouns" src="http://i.imgur.com/SC4XMHr.png"></a></p>
<p>Here's an auxiliary verb + "to be" detector ("will be", "I've always been", "has never been"):</p>
<p><a href="http://i.imgur.com/fRdatTW.png"><img alt="Modal Verbs" src="http://i.imgur.com/fRdatTW.png"></a></p>
<p>Here's a quote attributor:</p>
<p><a href="http://i.imgur.com/wQA8H0Q.png"><img alt="Quotes" src="http://i.imgur.com/wQA8H0Q.png"></a></p>
<p>There's even a MAGA and capitalization neuron:</p>
<p><a href="http://i.imgur.com/1QdT0MS.png"><img alt="MAGA" src="http://i.imgur.com/1QdT0MS.png"></a></p>
<p>And here are some of the proclamations the LSTM generates (okay, one of these is a real tweet):</p>
<p><a href="http://i.imgur.com/YSVVWGp.png"><img alt="Tweets" src="http://i.imgur.com/YSVVWGp.png"></a>
<a href="http://i.imgur.com/rhw4lTb.png"><img alt="Tweet" src="http://i.imgur.com/rhw4lTb.png"></a></p>
<p>Unfortunately, the LSTM merely learned to ramble like a madman.</p>
<h1>Recap</h1>
<p>That's it. To summarize, here's what you've learned:</p>
<p><img alt="Candidate Memory" src="http://i.imgur.com/WYVlc6w.png"></p>
<p>Here's what you should save:</p>
<p><img alt="Save" src="http://i.imgur.com/DqqJZAD.png"></p>
<p>And now it's time for that donut. </p>
<p>Thanks to <a href="https://github.com/crazydonkey200/tensorflow-char-rnn">Chen Liang</a> for some of the TensorFlow code I used, Ben Hamner and Kaggle for the <a href="https://www.kaggle.com/benhamner/clinton-trump-tweets">Trump dataset</a>, and, of course, Schmidhuber and Hochreiter for their <a href="http://www.bioinf.jku.at/publications/older/2604.pdf">original paper</a>. If you want to explore the LSTMs yourself, feel free to <a href="http://blog.echen.me/lstm-explorer">play around</a>!</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Moving Beyond CTR: Better Recommendations Through Human Evaluation2014-10-07T00:00:00+02:002014-10-07T00:00:00+02:00Edwin Chentag:blog.echen.me,2014-10-07:/2014/10/07/moving-beyond-ctr-better-recommendations-through-human-evaluation/<p>Imagine you're building a recommendation algorithm for your new online site. How do you measure its quality, to make sure that it's sending users relevant and personalized content? Click-through rate may be your initial hope…but after a bit of thought, it's not clear that it's the best metric after …</p><p>Imagine you're building a recommendation algorithm for your new online site. How do you measure its quality, to make sure that it's sending users relevant and personalized content? Click-through rate may be your initial hope…but after a bit of thought, it's not clear that it's the best metric after all.</p>
<p><strong>Take Google's search engine.</strong> In many cases, improving the quality of search results will <em>decrease</em> CTR! For example, the ideal scenario for queries like <a href="https://www.google.com/search?q=when%20was%20barack%20obama%20born">When was Barack Obama born?</a> is that users never have to click, since the question should be answered on the page itself.</p>
<p><strong>Or take Twitter, who one day might want to recommend you interesting tweets.</strong> Metrics like CTR, or even number of favorites and retweets, will probably optimize for showing quick one-liners and pictures of funny cats. But is a Reddit-like site what Twitter really wants to be? Twitter, for many people, started out as a news site, so users may prefer seeing links to deeper and more interesting content, even if they're less likely to click on each individual suggestion overall.</p>
<p><strong>Or take eBay, who wants to help you find the products you want to buy.</strong> Is CTR a good measure? Perhaps not: more clicks may be an indication that you're having trouble finding what you're looking for. What about revenue? That might not be ideal either: from the user perspective, you want to make your purchases at the lowest possible price, and by optimizing for revenue, eBay may be turning you towards more expensive products that make you a less frequent customer in the long run.</p>
<p><strong>And so on.</strong></p>
<p>So on many online sites, it's unclear how to measure the quality of personalization and recommendations using metrics like CTR, or revenue, or dwell time, or whatever. What's an engineer to do?</p>
<p>Well, consider the fact that many of these are <em>relevance</em> algorithms. Google wants to show you relevant search results. Twitter wants to show you relevant tweets and ads. Netflix wants to recommend you relevant movies. LinkedIn wants to find you relevant people to follow. <strong>So why, so often, do we never try to measure the relevance of our models?</strong></p>
<p>I'm a big fan of <a href="http://blog.echen.me/2013/01/08/improving-twitter-search-with-real-time-human-computation/"><em>man-in-the-machine</em> techniques</a>, so to get around this problem, I'm going to talk about a <strong>human evaluation</strong> approach to measuring the performance of personalization and discovery products. In particular, I'll use the example of related book suggestions on Amazon as I walk through the rest of this post.</p>
<h1>Amazon, and Moving Beyond Log-Based Metrics</h1>
<p>(Let's continue motivating a bit why log-based metrics are often imperfect measures of relevance and quality, as this is an important but difficult-to-understand point.)</p>
<p>So take Amazon's <em>Customers Who Bought This Item Also Bought</em> feature, which tries to show you related books.</p>
<p><a href="http://i.imgur.com/NoTD5Vk.png"><img alt="Amazon Related Books" src="http://i.imgur.com/NoTD5Vk.png"></a></p>
<p>To measure its effectiveness, the standard approach is to run a live experiment and measure the change in metrics like revenue or CTR.</p>
<p>But imagine we suddenly replace all of Amazon's book suggestions with pornography. What's going to happen? <a href="http://www.lazygamer.net/general-news/a-history-of-evony-ads-the-insanity-continues/"><em>CTR's likely to shoot through the roof!</em></a></p>
<p>Or suppose that we replace all of Amazon's related books with shinier, more expensive items. Again, CTR and revenue are likely to increase, as the flashier content draws eyeballs. But is this anything but a short-term boost? Perhaps the change decreases total sales in the long run, as customers start to find Amazon too expensive for their tastes and move to other marketplaces.</p>
<p>Scenarios like these are the machine learning analogue of turning ads into blinking marquees. While they might increase clicks and views initially, they're probably not optimizing user happiness or site quality for the future. So how can we avoid them, and ensure that the quality of our suggestions remains consistently high? <strong>This is a <em>related</em> books algorithm, after all – so why, by sticking to a live experiment and metrics like CTR, are we nowhere inspecting the relatedness of our recommendations?</strong></p>
<h1>Human Evaluation</h1>
<p>Solution: let's inject humans into the process. Computers can't measure relatedness (if they could, we'd be done), but people of course can.</p>
<p>For example, in the screenshot below, I asked a worker (on a crowdsourcing platform I've built on my own) to rate the first three <em>Customers Who Bought This Also Bought</em> suggestions for a book on <a href="http://www.barnesandnoble.com/">barnesandnoble.com</a>.</p>
<p><a href="http://i.imgur.com/XBE6tvc.png"><img alt="Fool's Assassin" src="http://i.imgur.com/XBE6tvc.png"></a></p>
<p>(Copying the text from the screenshot:</p>
<ul>
<li><strong><em>Fool's Assassin</em>, by Robin Hobb.</strong> (original book) "I only just read this recently, but it's one of my favorites of this year's releases. This book was so beautifully written. I've always been a fan of the Fantasy genre, but often times it's all stylistically similar. Hobb has a wonderful way with characters and narration. The story was absolutely heartbreaking and lead me wanting another book."</li>
<li><strong><em>The Broken Eye (Lightbringer Series #3)</em>, by Brent Weeks.</strong> (first related book) "It's a good, but not great, recommendation. This book sounds interesting enough. It's third in a series though, so I'd have to see how I like the first couple of books first."</li>
<li><strong><em>Dust and Light: A Sanctuary Novel</em>, by Carol Berg.</strong> (second related book) "It's an okay recommendation. I'm not so familiar with this author, but I like some of the premise of the book. I know the saying 'Don't judge a book by its cover...' but the cover art doesn't really appeal to me and kind of turns me off of the book."</li>
<li><strong><em>Tower Lord</em>, by Anthony Ryan.</strong> (third related book) "It's a good, but not great, recommendation. Another completely unfamiliar author to me (kind of like this though, it shows me new places to look for books!) This is also a sequel, though, so I'd have to see how I liked the first book before purchasing this one.")</li>
</ul>
<p>The recommendations are decent, but already we see a couple ways to improve them:</p>
<ul>
<li>First, two of the recommendations (<em>The Broken Eye</em> and <em>Tower Lord</em>) are each part of a series, but not Book #1. So one improvement would be to ensure that only series introductions get displayed unless they're followups to the main book.</li>
<li>Book covers matter! Indeed, the second suggestion looks more like a fantasy romance novel than the kind of fantasy that Robin Hobb tends to write. (So perhaps B&N should invest in some deep learning...)</li>
</ul>
<p>CTR and revenue certainly wouldn't give us this level of information, and it's not clear that they could even tell us our algorithms are producing irrelevant suggestions in the first place. Nowhere does the related scroll panel make it clear that two of the books are part of a series, so the CTR on those two books would be just as high as if they were indeed the series introductions. And if revenue is low, it's not clear whether it's because the suggestions are bad or because, separately, our pricing algorithms need improvement.</p>
<p>So in general, here's one way to understand the quality of a <em>Do This Next</em> algorithm:</p>
<ol>
<li>Take a bunch of items (e.g., books if we're Amazon or Barnes & Noble), and generate their related brethren.</li>
<li>Send these <item, related items> pairs off to a bunch of judges (say, by using a crowdsourcing platform like <a href="https://www.gethybrid.io">Hybrid</a>), and ask them to rate their relevance.</li>
<li>Analyze the data that comes back.</li>
</ol>
<h1>Algorithmic Relevance on Amazon, Barnes & Noble, and Google</h1>
<p>Let's make this process concrete. Pretend I'm a newly minted VP of What Customers Also Buy at Amazon, and I want to understand the product's flaws and stars.</p>
<p>I started by asking a couple hundred of my own human workers to take a book they enjoyed in the past year, and to find it on Amazon. They'd then take the first three related book suggestions from a different author, rate them on the following scale, and explain their ratings.</p>
<ul>
<li><strong>Great suggestion.</strong> I'd definitely buy it. (Very positive)</li>
<li><strong>Decent suggestion.</strong> I might buy it. (Mildly positive)</li>
<li><strong>Not a great suggestion.</strong> I probably wouldn't buy it. (Mildly negative)</li>
<li><strong>Terrible suggestion.</strong> I definitely wouldn't buy it. (Very negative)</li>
</ul>
<p>(Note: I usually prefer a three-point or five-point Likert scale with a "neutral" option, but I was feeling a little wild.)</p>
<p>For example, here's how a rater reviewed the related books for Anne Rice's <em>The Wolves of Midwinter</em>.</p>
<p><a href="http://i.imgur.com/eh9VKhR.png"><img alt="Anne Rice" src="http://i.imgur.com/eh9VKhR.png"></a></p>
<p>So how good are Amazon's recommendations? Quite good, in fact: 47% of raters said they'd definitely buy the first related book, another 29% said it was good enough that they might buy it, and only 24% of raters disliked the suggestion.</p>
<p><a href="http://i.imgur.com/MNSGpIi.png"><img alt="A1 Ratings" src="http://i.imgur.com/MNSGpIi.png"></a></p>
<p>The second and third book suggestions, while a bit worse, seem to perform pretty well too: around 65% of raters rated them positive.</p>
<p>What can we learn from the bad ratings? I ran a follow-up task that asked workers to categorize the bad related books, and here's the breakdown.</p>
<p><a href="http://i.imgur.com/h4DOvju.png"><img alt="Bad Suggestions Breakdown" src="http://i.imgur.com/h4DOvju.png"></a></p>
<ul>
<li><strong>Related but different-subtopic.</strong> These were book suggestions that were generally related to the original book, but that were in a different sub-topic that the rater wasn't interested in. For example, the first related book for Sun Tzu's <em>The Art of War</em> (a book nominally about war, but which nowadays has become more of a life hack book) was <a href="http://www.amazon.com/On-War-Carl-von-Clausewitz/dp/1469947021"><em>On War</em></a> (a war treatise), but the rater wasn't actually interested in the military aspects: <em>"I would not buy this book, because it only focuses on military war. I am not interested in that. I am interested in mental tactics that will help me prosper in life."</em></li>
<li><strong>Completely unrelated.</strong> These were book suggestions that were completely unrelated to the original book. For example, a Scrabble dictionary appearing on <em>The Art of War</em>'s page.</li>
<li><strong>Uninteresting.</strong> These were suggestions that were related, but whose storylines didn't appeal to the rater. <em>"The storyline doesn't seem that exciting. I am not a dog fan and it's about a dog."</em></li>
<li><strong>Wrong audience.</strong> These were book suggestions whose target audiences were quite different from the original book's audiences. In many cases, for example, a related book suggestion would be a children's book, but the original book would be geared towards adults. <em>"This seems to be a children's book. If I had a child I would definitely buy this; alas I do not, so I have no need for it."</em></li>
<li><strong>Wrong book type.</strong> Suggestions in this category were items like textbooks or appearing alongside novels.</li>
<li><strong>Disliked author.</strong> These recommendations were by similar authors, but one that the rater disliked. <em>"I do not like Amber Portwood. I would definitely not want to read a book by and about her."</em></li>
<li><strong>Not first in series.</strong> Some book recommendations would be for an interesting series the rater wasn't familiar with, but they wouldn't be the first book in the series.</li>
<li><strong>Bad rating.</strong> These were book suggestions that had a particularly low Amazon rating.</li>
</ul>
<p>So to improve their recommendations, Amazon could try improving its topic models, add age-based features to its books, distinguish between textbooks and novels, and invest in series detectors. (Of course, for all I know, they do all this already.)</p>
<h2>Competitive Analysis</h2>
<p>We now have a general grasp of Amazon's related book suggestions and how they could be improved, and just like we could quote a metric like a CTR of 6.2% or whatnot, we can also now quote a <em>relevance score</em> of 0.62 (or whatever). So let's turn to the question of how Amazon compares to other online booksellers like Barnes & Noble and Google Play.</p>
<p>I took the same task I used above, but this time asked raters to review the related suggestions on those two sites as well.</p>
<p><a href="http://i.imgur.com/kdWcSPs.png"><img alt="Amazon vs. B&N vs. Google" src="http://i.imgur.com/kdWcSPs.png"></a></p>
<p>In short,</p>
<ul>
<li><strong>Barnes & Nobles's algorithm are almost as good as Amazon's</strong>: the first three suggestions were rated positive 58% of the time, compared to 68% on Amazon.</li>
<li><strong>But Play Store recommendations are atrocious</strong> : a whopping 51% of Google's related book recommendations were marked terrible.</li>
</ul>
<p>Why are the Play Store's suggestions so bad? Let's look at a couple examples.</p>
<p>Here's the Play Store page for John Green's <em>The Fault in Our Stars</em>, a critics-loved-it book about cancer and romance (and now also a movie).</p>
<p><a href="http://i.imgur.com/oYKFLeV.png"><img alt="Fault in Our Stars" src="http://i.imgur.com/oYKFLeV.png"></a></p>
<p>Two of the suggestions are completely random: a poorly-rated Excel manual and a poorly-reviewed textbook on sexual health. The others are completely unrelated cowboy books, by a different John Green.</p>
<p>Here's the page for <em>The Strain</em>. In this case, all the suggestions are in a different language! And there are only four of them.</p>
<p><a href="http://i.imgur.com/F1XNxPz.png"><img alt="The Strain" src="http://i.imgur.com/F1XNxPz.png"></a></p>
<p>Once again asking raters to categorize all of the Play Store's bad recommendations...</p>
<p><a href="http://i.imgur.com/1rWRQ0r.png"><img alt="Google Atrocity" src="http://i.imgur.com/1rWRQ0r.png"></a></p>
<ul>
<li>45% of the time, the related book suggestions were completely unrelated to the original book in question. For example: displaying a physics textbook on the page for a romance novel.</li>
<li>32% of the time, there simply wasn't a book suggestion at all. (I'm guessing the Play Bookstore's catalog is pretty limited.)</li>
<li>14% of the time, the related books were in a different language.</li>
</ul>
<p>So despite Google's state-of-the-art machine learning elsewhere, its Play Store suggestions couldn't really get much worse.</p>
<h1>Side-by-Sides</h1>
<p>Let's step back a bit. So far I've been focusing on an <em>absolute</em> judgments paradigm, in which judges rate how relevant a book is to the original on an absolute scale. This model is great for understanding the overall quality of Amazon's related book algorithm.</p>
<p>In many cases, though, we want to use human evaluation to <em>compare</em> experiments. For example, it's common at many companies to:</p>
<ol>
<li>Launch a "human evaluation A/B test" before a live experiment, both to avoid accidentally sending out incredibly buggy experiments to users, as well as to avoid the long wait required in live tests.</li>
<li>Use a human-generated relevance score as a supplement to live experiment metrics when making launch decisions.</li>
</ol>
<p>For these kinds of tasks, what's preferable is often a <em>side-by-side</em> model, wherein judges are given two items and asked which one is better. After all, comparative judgments are often much easier to make than absolute ones, and we might want to detect differences at a finer level than what's available on an absolute scale.</p>
<p>The idea is that we can assign a score to each rating (negative, say, if the rater prefers the control item; positive if the rater prefers the experiment), and we aggregate these to form an overall score for the side-by-side. Then in much the same way that drops in CTR may block an experiment launch, a negative human evaluation score should also give much cause for concern.</p>
<p>Unfortunately, I don't have an easy way to generate data for a side-by-side (though I could perform a side-by-side on Amazon vs. Barnes & Noble), so I'll omit an example, but the idea should be pretty clear.</p>
<h1>Personalization</h1>
<p>Here's another subtlety. In my examples above, I asked raters to pick a starting book themselves (one that they read and loved in the past year), and then rate whether they personally would want to read the related suggestions.</p>
<p>Another approach is to pick the starting books <em>for</em> the raters, and then have the rate the related suggestions more objectively, by trying to put themselves in the shoes of someone who'd be interested in the starting item.</p>
<p>Which approach is better? As you can probably guess, there's no clear answer – it depends on the task and goals at hand.</p>
<p>Pros of the first approach:</p>
<ul>
<li>It's much more nuanced. It can often be hard to put yourself in someone else's shoes: would someone reading Harry Potter be interested in Twilight? On the one hand, they're both fantasy books; on the other hand, Twilight seems a bit more geared towards female audiences.</li>
</ul>
<p>Pros of the second approach:</p>
<ul>
<li>Sometimes, objectivity is a good thing. (Should you really care if someone dislikes Twilight simply because Edward reminds her of her boyfriend?)</li>
<li>Allowing people to choose their starting items may bias certain metrics. For instance, people are much more likely to choose popular books to rate, whereas we might want to measure the quality of Amazon's suggestions across a broader, more representative slice of its catalog.</li>
</ul>
<h1>Recap</h1>
<p>Let's review what I've discussed so far.</p>
<ol>
<li>Online, log-based metrics like CTR and revenue aren't necessarily the best measures of a discovery algorithm's quality. Items with high CTR, for example, may simply be racy and flashy, not the most relevant.</li>
<li>So instead of relying on these proxies, let's directly measure the relevance of our recommendations by asking a pool of human raters.</li>
<li>There are a couple different approaches for sending which items to be judged. We can let the raters choose the items (an approach which is often necessary for personalized algorithms), or we can generate the items ourselves (often useful for more objective tasks like search; this also has the benefit of making it easier to derive popularity-weighted or uniformly-weighted metrics of relevance). We can also take an absolute judgment approach, or use side-by-sides.</li>
<li>By analyzing the data from our evaluations, we can make better launch decisions, discover examples where our algorithms do very well or very poorly, and find patterns for improvement.</li>
</ol>
<p>What are some of the benefits and applications? </p>
<ol>
<li>As mentioned, log-based metrics like CTR and revenue don't always capture the signals we want, so human-generated scores of relevance (or other dimensions) are useful complements.</li>
<li>Human evaluations can make iteration quicker and easier. A algorithm change might normally require weeks of live testing before we gather enough data to know how it affects users, but we can easily have a task judged by humans in a couple hours.</li>
<li>Imagine we're an advertising company, and we choose which ads to show based on a combination of CTR and revenue. Once we gather hundreds of thousands of relevance judgments from our evaluations, we can build a <em>relevance</em>-based machine learning model to add to the mix, thereby injecting a more direct measure of quality into the system.</li>
<li>How can we decide what improvements need to be made to our models? Evaluations give us very concrete feedback and specific examples about what's working and what's wrong.</li>
</ol>
<p>In the spirit of item-item similarities, here are some other posts readers of this post might also want to read.</p>
<ul>
<li><a href="http://www.quora.com/How-does-Amazons-collaborative-filtering-recommendation-engine-work/answer/Edwin-Chen-1">How Amazon's related books feature works</a></li>
<li><a href="http://blog.echen.me/2013/01/08/improving-twitter-search-with-real-time-human-computation/">Real-time human computation at Twitter</a></li>
<li><a href="http://blog.echen.me/2012/02/09/movie-recommendations-and-more-via-mapreduce-and-scalding/">Generating movie recommendations in Hadoop and Scalding</a></li>
</ul>
<p>And finally, I'll end with a call for information. <strong>Do you run any evaluations, or use crowdsourcing platforms like Mechanical Turk or Crowdflower (whether for evaluation purposes or not)? Or do you want to?</strong> I'd love to talk to you to learn more about what you're doing, so please feel free to <a href="mailto:hello[at]echen.me">send me an email</a> and hit me up!</p>Propensity Modeling, Causal Inference, and Discovering Drivers of Growth2014-08-15T00:00:00+02:002014-08-15T00:00:00+02:00Edwin Chentag:blog.echen.me,2014-08-15:/2014/08/15/propensity-modeling-causal-inference-and-discovering-drivers-of-growth/<p>Imagine you just started a job at a new company. You watched <a href="http://youtu.be/AcNK7M2eCI4?t=1m5s">World War Z</a> recently, so you're in a skeptical mood, and given that your last two startups failed from what you believe to be a lack of data, you're giving everything an extra critical eye.</p>
<p>You start by …</p><p>Imagine you just started a job at a new company. You watched <a href="http://youtu.be/AcNK7M2eCI4?t=1m5s">World War Z</a> recently, so you're in a skeptical mood, and given that your last two startups failed from what you believe to be a lack of data, you're giving everything an extra critical eye.</p>
<p>You start by thinking about the impact of the sales team. <em>How much extra revenue are they generating for the company?</em> The sales folks you've met say that over 90% of the leads they've talked to end up buying the company's product – but, you wonder, how many of those leads would have converted anyways?</p>
<p>You take a look at the logs, and notice something interesting: last week was hack week, and half the salesforce took time off from making calls to make Marauder's Maps instead, yet the rate of converted leads remained the same...*</p>
<p>Suddenly, one of your teammates drops by your desk. He's making a batch of <a href="http://www.nytimes.com/2014/05/29/technology/personaltech/the-soylent-revolution-will-not-be-pleasurable.html">Soylent</a>, and he wants you to take a sip. It looks nasty, so you ask what the benefits are, and he responds that his friends who've been drinking it for the past few months just ran a marathon! <em>Oh, did they just start running?</em> Nope, they ran the marathon last year too!...</p>
<p>*Inspired by a true story.</p>
<h1>Causal Inference</h1>
<p>Causality is incredibly important, yet often extremely difficult to establish.</p>
<p>Do patients who self-select into taking a new drug get better <em>because</em> the drug works, or would they have gotten better anyways? Is your salesforce actually effective, or are they simply talking to the customers who already plan to convert? Is Soylent (or your company's million-dollar ad campaign) truly worth your time?</p>
<p>In an ideal world, we'd be able to run experiments – the gold standard for measuring causality – whenever we wish. In the real world, however, we can't. There are ethical qualms with giving certain patients placebos, or dangerous and untested drugs. Management may be unwilling to take a potential short-term revenue hit by assigning sales to random customers, and a team earning commission-based bonuses may rebel against the thought as well.</p>
<p>How can we understand causal lifts in the absence of an A/B test? This is where propensity modeling, or other techniques of causal inference, comes into play.</p>
<h1>Propensity Modeling</h1>
<p>So suppose we want to model the effect of drinking Soylent using a propensity model technique. To explain the idea, let's start with a thought experiment. </p>
<p>Imagine Brad Pitt has a twin brother, indistinguishable in every way: Brad 1 and Brad 2 wake up at the same time, they eat the same foods, they exercise the same amount, and so on. One day, Brad 1 happens to receive the last batch of Soylent from a marketer on the street, while Brad 2 does not, and so only Brad 1 begins to incorporate Soylent into his diet. In this scenario, any subsequent difference in behavior between the twins is precisely the drink's effect.</p>
<p>Taking this scenario into the real world, one way to estimate the Soylent's effect on health would be as follows:</p>
<ul>
<li>For every Soylent drinker, find a Soylent abstainer who's as close a match as possible. For example, we might match a Soylent-drinking Jay-Z with a non-Soylent Kanye, a Soylent-drinking Natalie Portman with a non-Soylent Keira Knightley, and a Soylent-drinking JK Rowling with a non-Soylent Stephenie Meyer.</li>
<li>We measure Soylent's effect as the difference between each twin pair.</li>
</ul>
<p>However, finding closely matching twins is extremely difficult in practice. Is Jay-Z really a close match with Kanye, if Jay-Z sleeps one hour more on average? What about the Jonas Brothers and One Direction?</p>
<p><strong>Propensity modeling</strong>, then, is a simplification of this twin matching procedure. Instead of matching pairs of people based on all the variables we have, we simply match all users based on a single number, the likelihood ("propensity") that they'll start to drink Soylent.</p>
<p>In more detail, here's how to build a propensity model.</p>
<ul>
<li>First, select which variables to use as features. (e.g., what foods people eat, when they sleep, where they live, etc.)</li>
<li>Next, build a probabilistic model (say, a logistic regression) based on these variables to predict whether a user will start drinking Soylent or not. For example, our training set might consist of a set of people, some of whom ordered Soylent in the first week of March 2014, and we would train the classifier to model which users become the Soylent users.</li>
<li>The model's probabilistic estimate that a user will start drinking Soylent is called a <strong>propensity score</strong>.</li>
<li>Form some number of buckets, say 10 buckets in total (one bucket covers users with a 0.0 - 0.1 propensity to take the drink, a second bucket covers users with a 0.1 - 0.2 propensity, and so on), and place people into each one.</li>
<li>Finally, compare the drinkers and non-drinkers within each bucket (say, by measuring their subsequent physical activity, weight, or whatever measure of health) to estimate Soylent's causal effect.</li>
</ul>
<p>For example, here's a hypothetical distribution of Soylent and non-Soylent ages. We see that drinkers tend to be quite a bit older, and this confounding fact is one reason we can't simply run a correlational analysis.</p>
<p><a href="http://i.imgur.com/vQQACqf.png"><img alt="Age Distribution" src="http://i.imgur.com/vQQACqf.png"></a></p>
<p>After training a model to estimate Soylent propensity and group users into propensity buckets, this might be a graph of the effect that Soylent has on a person's weekly running mileage.</p>
<p><a href="http://i.imgur.com/GwxpWve.png"><img alt="Soylent Effect" src="http://i.imgur.com/GwxpWve.png"></a></p>
<p>In the above (hypothetical) graph, each row represents a propensity bucket of people, and the exposure week denotes the first week of March, when the treatment group received their Soylent shipment. We see that prior to that week, both groups of people track quite well. After the treatment group (the Soylent drinkers) start their plan, however, their weekly running mileage ramps up, which forms our estimate of the drink's causal effect.</p>
<h1>Other Methods of Causal Inference</h1>
<p>Of course, there are many other methods of causal inference on observational data. I'll run through two of my favorites. (I originally wrote this post in response to a question on Quora, which is why I take my examples from there.)</p>
<h2>Regression Discontinuity</h2>
<p>Quora recently started displaying badges of status on the profiles of its <a href="http://blog.quora.com/Announcing-Top-Writers-2013">Top Writers</a>, so suppose we want to understand the effect of this feature. (Assume that it's impossible to run an A/B test now that the feature has been launched.) Specifically, does the badge itself cause users to gain more followers?</p>
<p>For simplicity, let's assume that the badge was given to all users who received at least 5000 upvotes in 2013. The idea behind a <strong>regression discontinuity design</strong> is that the difference between those users who just barely receive a Top Writer badge (i.e., receive 5000 upvotes) and those who just barely don't (i.e., receive 4999 upvotes) is more or less random chance, so we can use this threshold to estimate a causal effect.</p>
<p>For example, in the imaginary graph below, the discontinuity at 5000 upvotes suggests that a Top Writer badge leads to around 100 more followers on average.</p>
<p><a href="http://i.imgur.com/7Mprryq.png"><img alt="Regression Discontinuity" src="http://i.imgur.com/7Mprryq.png"></a></p>
<h2>Natural Experiments</h2>
<p>Understanding the effect of a Top Writer badge is a fairly uninteresting question, though. (It just makes an easy example.) A deeper, more fundamental question could be to ask: what happens when a user discovers a new writer that they love? Does the writer inspire them to write some of their own content, to explore more of the same topics, and through curation lead them to engage with the site even more? How important, in other words, is the connection to a great <em>user</em> as opposed to the reading of random, great <em>posts</em>?</p>
<p>I studied an analogous question when I was at Google, so instead of making up an imaginary Quora case study, I'll describe some of that work here.</p>
<p>So let's suppose we want to understand what would happen if we were able to match users to the perfect YouTube channel. How much is the ultimate recommendation worth?</p>
<ul>
<li>Does falling in love with a new channel lead to engagement above and beyond activity on the channel itself, perhaps because users return to YouTube specifically for the new channel and stay to watch more? (a <strong>multiplicative</strong> effect) In the TV world, for example, perhaps many people stay at home on Sunday nights <em>specifically</em> to catch the latest episode of Real Housewives, and channel surf for even more entertainment once it's over.</li>
<li>Does falling in love with a new channel simply increase activity on the channel alone? (an <strong>additive</strong> effect)</li>
<li>Does a new channel replace existing engagement on YouTube? After all, maybe users only have a limited amount of time they can spend on the site. (a <strong>neutral</strong> effect)</li>
<li>Does the perfect channel actually cause users to spend less time overall on the site, since maybe they spend less time idly browsing and channel surfing once they have concrete channels they know how to turn to? (a <strong>negative</strong> effect)</li>
</ul>
<p>As always, an A/B test would be ideal, but it's impossible in this case to run: we can't force users to fall in love with a channel (we can recommend them channels, but there's no guarantee they'll actually like them), and we can't forcibly block them from certain channels either.</p>
<p>One approach is to use a <strong>natural experiment</strong> (a scenario in which the universe itself somehow generates a random-like assignment) to study this effect. Here's the idea.</p>
<p>Consider a user who uploads a new video every Wednesday. One month, he lets his subscribers know that he won’t be uploading any new videos for a few weeks, while he goes on vacation.</p>
<p>How do his subscribers respond? Do they stop watching YouTube on Wednesdays, since his channel was the sole reason for their visits? Or is their activity relatively unaffected, since they only watch his content when it appears on the front page?</p>
<p>Imagine, instead, that the channel starts uploading a new video every Friday. Do his subscribers start to visit then as well? And now that they're on YouTube, do they merely stay for the new video, or does their visit lead to a sinkhole of searches and related content too?</p>
<p>As it turns out, these scenarios do happen. For example, here's a calendar of when one popular channel uploads videos. You can see that in 2011, it tended to upload videos on Tuesdays and Fridays, but it shifted to uploads on Wednesday and Saturday at the end of the year.</p>
<p><a href="http://i.imgur.com/wCiHHAD.png"><img alt="Uploads Calendar" src="http://i.imgur.com/wCiHHAD.png"></a></p>
<p>By using this shift as natural experiment that "quasi-randomly" removes a well-loved channel on certain days and introduces it on others, we can try to understand the effect of successfully making the perfect recommendation.</p>
<p>(This is probably a somewhat convoluted example of a natural experiment. For an example that perhaps illustrates the idea more clearly, suppose we want to understand the effect of income on mental health. We can't force some people to become poor or rich, and a correlational study is clearly flawed. <a href="http://opinionator.blogs.nytimes.com/2014/01/18/what-happens-when-the-poor-receive-a-stipend/">This NY Times article</a> describes a natural experiment when a group of Cherokee Indians distributed casino profits to its members, thereby "randomly" lifting some of them out of poverty.</p>
<p>Another example, assuming there's nothing special about the period in which hack week occurs, is the use of hack week as an instrument that quasi-randomly "prevents" the sales team from doing their job, as in the scenario I described above.)</p>
<h1>Discovering Drivers of Growth</h1>
<p>Let's go back to propensity modeling.</p>
<p>Imagine that we're on our company's Growth team, and we're tasked with figuring out how to turn casual users of the site into users that return every day. What do we do?</p>
<p>The propensity modeling approach might be the following. We could take a list of features (installing the mobile app, logging in, signing up for a newsletter, following certain users, etc.), and build a propensity model for each one. We could then rank each feature by its estimated causal effect on engagement, and use the ordered list of features to prioritize our next sprint. (Or we could use these numbers in order to convince the exec team that we need more resources.) This is a slightly more sophisticated version of the idea of building an engagement regression model (or a churn regression model), and examining the weights on each feature.</p>
<p>Despite writing this post, though, I admit I'm generally not a fan of propensity modeling for many applications in the tech world. (I haven't worked in the medical field, so I don't have a strong opinion on its usefulness there, though I think it's a little more necessary there.) I'll save more of my reasons for another time, but after all, causal inference is extremely difficult, and we're never going to be able to control for all the hidden influencers that can bias a treatment. Also, the mere fact that we have to choose which features to include in our model (and remember: building features is very time-consuming and difficult) means that we already have a strong prior belief on the usefulness of each feature, whereas what we'd really like to do is to discover hidden motivations of engagement that we've never thought of.</p>
<p>So what can we do instead?</p>
<p><strong>If we're trying to understand what drives people to become heavy users of the site, why don't we simply ask them?</strong></p>
<p>In more detail, let's do the following:</p>
<ul>
<li>First, we'll run a survey on a couple hundred of users.</li>
<li>In the survey, we'll ask them whether their engagement on the site has increased, decreased, or remained about the same over the past year. We'll also ask them to explain possible reasons for their change in activity, and to describe how they use the site currently. We can also ask for supplemental details, like their demographic information.</li>
<li>Finally, we can filter all responses for those users who heavily increased their engagement over the past year (or who heavily decreased it, if we're trying to understand churn), and analyse their responses for the reasons.</li>
</ul>
<p>For example, here's one interesting response I got when I ran this study at YouTube.</p>
<p><em>"I have always been a big music fan, but recently I took up playing the guitar. Because of my new found passion (playing the guitar) my desire to watch concerts has increased. I started watching a whole lot of music festivals and concerts that are posted on Youtube and other music videos. I have spent a lot of time also watching guitar lessons on Youtube (from www.justinguitar.com)."</em></p>
<p>This response was representative of a general theme the survey uncovered: one big driver of engagement seemed to come from people discovering a new offline hobby, and using YouTube to increase their appreciation of it. People who wanted to start cooking at home would turn to YouTube for recipe videos, people who started playing tennis or some other sport would go to YouTube for lessons or other great shots, college students would look for channels like Khan Academy to supplement their lectures, and so on. In other words, offline activities were driving online growth, and instead of trying to figure out what kinds of <em>online</em> content people were interested in (which articles did they like on Facebook, who did they follow on Twitter, what did they read on Reddit), perhaps we should have been focusing on bringing their physical hobbies into the digital world.</p>
<p>This "offline hobby" idea certainly wouldn't have been a feature I would have thrown into any engagement model, even if only because it's a very difficult feature to create. (How do we know which videos are related to real-world behavior?) But now that we suspect it's a potentially big driver of growth ("potentially" because, of course, surveys aren't necessarily representative), it's something we can spend a lot more time studying in the logs.</p>
<h1>End</h1>
<p>To summarize: propensity modeling is a powerful technique for measuring causal effects in the absence of a randomized experiment. </p>
<p>Purely correlational analyses on top of observational studies can be very dangerous, after all. To take my favorite example: if we find that cities with more policemen tend to have more crime, does this mean that we should try to reduce the size of our police forces in order to reduce the nation's amount of crime?</p>
<p>For another example, <a href="http://andrewgelman.com/2005/01/07/could_propensit/">here</a>'s a post by Gelman on contradictory conclusions about hormone replacement therapy in the <a href="http://andrewgelman.com/2005/01/07/could_propensit/">Harvard Nurses Study</a>.</p>
<p>That said, remember that (as always) a model is only as good as the data that you feed it. It's super difficult to account for all the hidden variables that might matter, and what you think might be a well-designed causal model might well in fact be missing many hidden factors. (I actually remember hearing that a propensity model on the nurses study generated a flawed conclusion, though I can't find any references to this at the moment.) So consider whether there are other approaches you can take, whether it's an easier-to-understand causal technique or even just asking your users, and even if a randomized experiment seems too difficult to run now, the effort may be well worth the trouble in the end.</p>Product Insights for Airbnb2014-08-14T00:00:00+02:002014-08-14T00:00:00+02:00Edwin Chentag:blog.echen.me,2014-08-14:/2014/08/14/product-insights-for-airbnb/<p>I love studying users and products, and think data science can be extremely useful in guiding product/strategy as a whole. So I thought it would be fun to depart from the usual machine learning and engineering things I write about, and do a quick study of Airbnb.</p>
<p>Think of …</p><p>I love studying users and products, and think data science can be extremely useful in guiding product/strategy as a whole. So I thought it would be fun to depart from the usual machine learning and engineering things I write about, and do a quick study of Airbnb.</p>
<p>Think of this like business analysis, or strategy – from a data science point of view. </p>
<p>(It's in <a href="https://speakerdeck.com/edwinchen/product-insights-for-airbnb">slide deck form</a>, of course, because that's how these things roll.)</p>
<script async class="speakerdeck-embed" data-id="228abf8d857a4a21b9e79b4b518f507d" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script>Improving Twitter Search with Real-Time Human Computation2013-01-08T04:15:00+01:002013-01-08T04:15:00+01:00Edwin Chentag:blog.echen.me,2013-01-08:/2013/01/08/improving-twitter-search-with-real-time-human-computation
<div class="entry-content"><p><em>(This is a post from the <a href="http://engineering.twitter.com/2013/01/improving-twitter-search-with-real-time.html">Twitter Engineering Blog</a> that I wrote with <a href="https://twitter.com/alpa">Alpa Jain</a>.)</em></p>
<p>One of the magical things about Twitter is that it opens a window to the world in <strong>real-time</strong>. An event happens, and just seconds later, it’s shared for people across the planet to see …</p></div>
<div class="entry-content"><p><em>(This is a post from the <a href="http://engineering.twitter.com/2013/01/improving-twitter-search-with-real-time.html">Twitter Engineering Blog</a> that I wrote with <a href="https://twitter.com/alpa">Alpa Jain</a>.)</em></p>
<p>One of the magical things about Twitter is that it opens a window to the world in <strong>real-time</strong>. An event happens, and just seconds later, it’s shared for people across the planet to see.</p>
<p>Consider, for example, what happened when Flight 1549 crashed in the Hudson.</p>
<blockquote class="twitter-tweet"><p><a href="http://twitpic.com/135xa">http://twitpic.com/135xa</a> - There’s a plane in the Hudson. I’m on the ferry going to pick up the people. Crazy.</p>— Janis Krums (@jkrums) <a href="https://twitter.com/jkrums/status/1121915133" data-datetime="2009-01-15T20:36:04+00:00">January 15, 2009</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<br />
<p>When Osama bin Laden was killed.</p>
<blockquote class="twitter-tweet"><p>Helicopter hovering above Abbottabad at 1AM (is a rare event).</p>— Sohaib Athar (@ReallyVirtual) <a href="https://twitter.com/ReallyVirtual/status/64780730286358528" data-datetime="2011-05-01T19:58:24+00:00">May 1, 2011</a></blockquote>
<br />
<p>Or when Mitt Romney mentioned binders during the presidential debates.</p>
<blockquote class="twitter-tweet"><p>Boy, I’m full of women! <a href="https://twitter.com/search/%23debates">#debates</a></p>— Romney’s Binder (@RomneysBinder) <a href="https://twitter.com/RomneysBinder/status/258383626918588417" data-datetime="2012-10-17T01:47:11+00:00">October 17, 2012</a></blockquote>
<br />
<p>When each of these events happened, people instantly came to Twitter – and, in particular, Twitter search – to discover what was happening.</p>
<p>From a search and advertising perspective, however, these sudden events pose several challenges:</p>
<ol>
<li>The queries people perform have never before been seen, so it’s impossible to know beforehand what they mean. How would you know that #bindersfullofwomen refers to politics, and not office accessories, or that people searching for “Horses and Bayonets” are interested in the debates?</li>
<li>Since these spikes in search queries are so <a href="http://arxiv.org/abs/1205.6855">short-lived</a>, there’s only a short window of opportunity to learn what they mean.</li>
</ol>
<p>So an event happens, people instantly come to Twitter to search for the event, and we need to teach our systems what these queries mean as quickly as we can, because in just a few hours those searches will be gone.</p>
<p>How do we do this? We’ll describe a novel real-time human computation engine we built that allows us to find search queries as soon as they’re trending, send these queries to real humans to be judged, and finally incorporate these human annotations into our backend models.</p>
<h2>Overview</h2>
<p>Before we dive into the details, here’s an overview of how the system works.</p>
<p>(1) First, we monitor for which search queries are currently popular.</p>
<p>Behind the scenes: we run a Storm topology that tracks statistics on search queries.</p>
<p>For example: the query “Big Bird” may be averaging zero searches a day, but at 6pm on October 3, we suddenly see a spike in searches from the US.</p>
<p>(2) Next, as soon as we discover a new popular search query, we send it to our human evaluation systems, where judges are asked a variety of questions about the query.</p>
<p>Behind the scenes: when the Storm topology detects that a query has reached sufficient popularity, it connects to a Thrift API that dispatches the query to my team's human computation platform, and then polls for a response.</p>
<p>For example: as soon as we notice “Big Bird” spiking, we may ask human judges to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps us serve relevant tweets and ads.</p>
<p>Finally, after a response from a judge is received, we push the information to our backend systems, so that the next time a user searches for a query, our machine learning models will make use of the additional information. For example, suppose our human judges tell us that “Big Bird” is related to politics; the next time someone performs this search, we know to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.</p>
<p>Let’s now explore the first two sections above in more detail.</p>
<h2>Monitoring for popular queries</h2>
<p><a href="https://github.com/nathanmarz/storm">Storm</a> is a distributed system for real-time computation. In contrast to <em>batch</em> systems like Hadoop, which often introduce delays of hours or more, Storm allows us to run online data processing algorithms to discover search spikes as soon as they happen.</p>
<p>In brief, running a job on Storm involves creating a Storm topology that describes the processing steps that must occur, and deploying this topology to a Storm cluster. A topology itself consists of three things:</p>
<ul>
<li><strong>Tuple streams</strong> of data. In our case, these may be tuples of (search query, timestamp).</li>
<li><strong>Spouts</strong> that produce these tuple streams. In our case, we attach spouts to our search logs, which get written to every time a search occurs.</li>
<li><strong>Bolts</strong> that process tuple streams. In our case, we use bolts for operations like updating total query counts, filtering out non-English queries, and checking whether an ad is currently being served up for the query.</li>
</ul>
<p>Here’s a step-by-step walkthrough of how our popular query topology works:</p>
<ol>
<li>Whenever you perform a search on Twitter, the search request gets logged to a <a href="http://kafka.apache.org/">Kafka queue</a>.</li>
<li>The Storm topology attaches a spout to this Kafka queue, and the spout emits a tuple containing the query and other metadata (e.g., the time the query was issued and its location) to a bolt for processing.</li>
<li>This bolt updates the count of the number of times we’ve seen this query, checks whether the query is “currently popular” (using various statistics like time-decayed counts, the geographic distribution of the query, and the last time this query was sent for annotations), and dispatches it to our human computation pipeline if so.</li>
</ol>
<p>One interesting feature of our popularity algorithm is that we often rejudge queries that have been annotated before, since the intent of a search can change. For example, perhaps people normally search for “Clint Eastwood” because they’re interested in his movies, but during the Republican National Convention users may have wanted to see tweets that were more political in nature.</p>
<h2>Human evaluation of popular search queries</h2>
<p>At Twitter, we use human computation for a variety of tasks. (See also <a href="https://github.com/twitter/clockworkraven">Clockwork Raven</a>, an open-source crowdsourcing platform we built that makes launching tasks easier.) For example, we often run experiments to measure ad relevance and search quality, we use it to gather data to train and evaluate our machine learning models, and in this section we’ll describe how we use it to boost our understanding of popular search queries.</p>
<p>So suppose that our Storm topology has detected that the query “Big Bird” is suddenly spiking. Since the query may remain popular for only a few hours, we send it off to live humans, who can help us quickly understand what it means; this dispatch is performed via a Thrift service that allows us to design our tasks in a <a href="http://engineering.twitter.com/2012/08/crowdsourced-data-analysis-with.html">web frontend</a>, and later programmatically submit them to our human computation platform using any of the different languages we use across Twitter.</p>
<p>On our crowdsourcing platforms, judges are asked several questions about the query that help us serve better ads. Without going into the exact questions, here are flavors of a few possibilities:</p>
<ul>
<li>What category does the query belong to? For example, “Stanford” may typically be an education-related query, but perhaps there’s a football game between Stanford and Berkeley at the moment, in which case the current search intent would be sports.</li>
<li>Does the query refer to a person? If so, who, and what is their Twitter handle if they have one? For example, the query “Happy Birthday Harry” may be trending, but it’s hard to know beforehand which of the numerous celebrities named Harry it’s referring to. Is it <a href="https://twitter.com/onedirection">One Direction</a>’s <a href="https://twitter.com/Harry_Styles">Harry Styles</a>, in which case the searcher is likely to be interested in teen pop? Harry Potter, in which case the searcher is likely to be interested in fantasy novels? Or someone else entirely?</li>
</ul>
<h3>Labelers in the machine</h3>
<p>Since humans are core to this system, let’s describe how our workforce was designed to give us fast, reliable results.</p>
<p>For completing all our tasks, we use a <em>custom</em> pool of judges to ensure high quality. Other typical possibilities in the crowdsourcing world are to use a static set of in-house judges, to use the standard worker filters that Amazon provides, or to go through an outside company like <a href="http://crowdflower.com/">Crowdflower</a>. We’ve experimented with these other solutions, and while they have their own benefits, we found that a custom pool fit our needs best for a few reasons:</p>
<ul>
<li>In-house judges can provide high-quality work as well, but they usually work standard hours (for example, 9 to 5 if they work onsite, or a relatively fixed and limited set of hours if they work from home), it can be difficult to communicate with them and schedule them for work, and it’s hard to scale the hiring of more judges.</li>
<li>Using Crowdflower or Amazon’s standard filters makes it easy to scale the workforce, but their trust algorithms aren’t perfect, so an endless problem is that spammy workers get through and many of the judgments will be very poor quality. Two methods of combatting low quality are to seed gold standard examples for which you know the true response throughout your task, or to use statistical analysis to determine which workers are the good ones, but these can be time-consuming and expensive to create, and we often run tasks of a free-response researchy nature for which these solutions don’t work. Another problem is that using these filters gives you a <em>fluid</em>, constantly changing set of workers, which makes them hard to train.</li>
</ul>
<p>In contrast:</p>
<ul>
<li>Our custom pool of judges work virtually all day. For many of them, this is a full-time job, and they’re geographically distributed, so our tasks complete quickly at all hours; we can easily ask for thousands of judgments before lunch, and have them finished by the time we get back, which makes iterating on our experiments much easier.</li>
<li>We have several forums, mailing lists, and even live chatrooms set up, all of which makes it easy for judges to ask us questions and to respond to feedback. Our judges will even give <em>us</em> suggestions on how to improve our tasks; for example, when we run categorization tasks, they’ll often report helpful categories that we should add.</li>
<li>Since we only launch tasks on demand, our judges are never idly twiddling their thumbs waiting for tasks or completing busywork, and our jobs are rarely backlogged.</li>
<li>Because our judges are culled from the best of the crowdsourcing world, they’re experts at the kinds of tasks we send, and can often provide higher quality at a faster rate than what even in-house judges provide. For example, they’ll often use the forums and chatrooms to collaborate amongst themselves to give us the best judgments, and they’re already familiar with the Firefox and Chrome scripts that help them be the most efficient at their work.</li>
</ul>
<p>All the benefits described above are especially valuable in this real-time search annotation case:</p>
<ul>
<li>Having highly trusted workers means we don’t need to wait for multiple annotations on a single search query to confirm validity, so we can send responses to our backend as soon as a single judge responds. This entire pipeline is design for <em>real-time</em>, after all, so the lower the latency on the human evaluation part, the better.</li>
<li>The static nature of our custom pool means that the judges are already familiar with our questions, and don’t need to be trained again.</li>
<li>Because our workers aren’t limited to a fixed schedule or location, they can work anywhere, anytime – which is a requirement for this system, since global event spikes on Twitter are not beholden to a 9-to-5.</li>
<li>And with the multiple easy avenues of communication we have set up, it’s easy for us to answer questions that might arise when we add new questions or modify existing ones.</li>
</ul>
</div>
Edge Prediction in a Social Graph: My Solution to Facebook's User Recommendation Contest on Kaggle2012-07-31T04:15:00+02:002012-07-31T04:15:00+02:00Edwin Chentag:blog.echen.me,2012-07-31:/2012/07/31/edge-prediction-in-a-social-graph-my-solution-to-facebooks-user-recommendation-contest-on-kaggle/
<div class="entry-content"><p>A couple weeks ago, Facebook launched a <a href="http://www.kaggle.com/c/FacebookRecruiting/">link prediction contest</a> on Kaggle, with the goal of recommending missing edges in a social graph. <a href="http://blog.echen.me/2011/09/07/information-transmission-in-a-social-network-dissecting-the-spread-of-a-quora-post/">I love investigating social networks</a>, so I dug around a little, and since I did well enough to score one of the coveted prizes, I’ll share …</p></div>
<div class="entry-content"><p>A couple weeks ago, Facebook launched a <a href="http://www.kaggle.com/c/FacebookRecruiting/">link prediction contest</a> on Kaggle, with the goal of recommending missing edges in a social graph. <a href="http://blog.echen.me/2011/09/07/information-transmission-in-a-social-network-dissecting-the-spread-of-a-quora-post/">I love investigating social networks</a>, so I dug around a little, and since I did well enough to score one of the coveted prizes, I’ll share my approach here.</p>
<p>(For some background, the contest provided a training dataset of edges, a test set of nodes, and contestants were asked to predict missing outbound edges on the test set, using mean average precision as the evaluation metric.)</p>
<h1>Exploration</h1>
<p>What does the network look like? I wanted to play around with the data a bit first just to get a rough feel, so I made an <a href="http://link-prediction.herokuapp.com/network">app</a> to interact with the network around each node.</p>
<p>Here’s a sample:</p>
<p><a href="http://link-prediction.herokuapp.com/network"><img src="http://i.imgur.com/L2S5GC1.png" alt="1 Untrimmed Network" /></a></p>
<p>(Go ahead, click on the picture to <a href="http://link-prediction.herokuapp.com/network">play with the app yourself</a>. It’s pretty fun.)</p>
<p>The node in black is a selected node from the training set, and we perform a breadth-first walk of the graph out to a maximum distance of 3 to uncover the local network. Nodes are sized according to their distance from the center, and colored according to a chosen metric (a personalized PageRank in this case; more on this later).</p>
<p>We can see that the central node is friends with three other users (in red), two of whom have fairly large, disjoint networks.</p>
<p>There are quite a few dangling nodes (nodes at distance 3 with only one connection to the rest of the local network), though, so let’s remove these to reveal the core structure:</p>
<p><a href="http://link-prediction.herokuapp.com/network"><img src="http://i.imgur.com/w1kAj6x.png" alt="1 Untrimmed Network" /></a></p>
<p>And here’s an embedded version you can manipulate inline:</p>
<iframe width="600px" height="500px" src="http://link-prediction.herokuapp.com/network?for_embed=true"></iframe>
<p>Since the default view doesn’t encode the distinction between following and follower relationships, we can mouse over each node to see who it follows and who it’s followed by. Here, for example, is the following/follower network of one of the central node’s friends:</p>
<p><a href="http://i.imgur.com/38laphs.png"><img src="http://i.imgur.com/38laphs.png" alt="1 - Friend1" /></a></p>
<p>The moused over node is highlighted in black, its friends (users who both follow the node and are followed back in turn) are colored in purple, its followees are teal, and its followers in orange. We can also see that the node shares a friend with the central user (<a href="http://en.wikipedia.org/wiki/Triadic_closure">triadic closure</a>, <em>holla!</em>).</p>
<p>Here’s another network, this time of the friend at the bottom:</p>
<p><a href="http://i.imgur.com/bmJdrGT.png"><img src="http://i.imgur.com/bmJdrGT.png" alt="1 - Friend2" /></a></p>
<p>Interestingly, while the first friend had several only-followers (in orange), the second friend has none. (which suggests, perhaps, a node-level feature that measures how follow-hungry a user is…)</p>
<p>And here’s one more node, a little further out (maybe a celebrity, given it has nothing but followers?):</p>
<p><a href="http://i.imgur.com/vXXT1np.png"><img src="http://i.imgur.com/vXXT1np.png" alt="1 - Celebrity" /></a></p>
<h2>The Quiet One</h2>
<p>Let’s take a look at another graph, one whose local network is a little smaller:</p>
<p><a href="http://i.imgur.com/YDjamZK.png"><img src="http://i.imgur.com/YDjamZK.png" alt="4 Network" /></a></p>
<h2>A Social Butterfly</h2>
<p>And one more, whose local network is a little larger:</p>
<p><a href="http://i.imgur.com/0dPnb3V.png"><img src="http://i.imgur.com/0dPnb3V.png" alt="2 Network" /></a></p>
<p><a href="http://i.imgur.com/RSS8Ikh.png"><img src="http://i.imgur.com/RSS8Ikh.png" alt="2 Network - Friend" /></a></p>
<p>Again, I encourage everyone to play around with the app <a href="http://link-prediction.herokuapp.com/network">here</a>, and I’ll come back to the question of coloring each node later.</p>
<h1>Distributions</h1>
<p>Next, let’s take a more quantitative look at the graph.</p>
<p>Here’s the distribution of the number of followers of each node in the training set (cut off at 50 followers for a better fit – the maximum number of followers is 552), as well as the number of users each node is following (again, cut off at 50 – the maximum here is 1566)</p>
<p><a href="http://i.imgur.com/c5BfwaG.png"><img src="http://i.imgur.com/c5BfwaG.png" alt="Training Followers" /></a></p>
<p><a href="http://i.imgur.com/NLBHgy8.png"><img src="http://i.imgur.com/NLBHgy8.png" alt="Training Followees" /></a></p>
<p>Nothing terribly surprising, but that alone is good to verify. (For people tempted to mutter about power laws, I’ll hold you off with the bitter coldness of <a href="http://cscs.umich.edu/~crshalizi/weblog/491.html">baby Gauss’s tears</a>.)</p>
<p>Similarly, here are the same two graphs, but limited to the nodes in the test set alone:</p>
<p><a href="http://i.imgur.com/O6ogEYy.png"><img src="http://i.imgur.com/O6ogEYy.png" alt="Test Followers" /></a></p>
<p><a href="http://i.imgur.com/roUrV5I.png"><img src="http://i.imgur.com/roUrV5I.png" alt="Test Followees" /></a></p>
<p>Notice that there are relatively more test set users with 0 followees than in the full training set, and relatively fewer test set users with 0 followers. This information could be used to better simulate a validation set for model selection, though I didn’t end up doing this myself.</p>
<h1>Preliminary Probes</h1>
<p>Finally, let’s move on to the models themselves.</p>
<p>In order to quickly get up and running on a couple prediction algorithms, I started with some unsupervised approaches. For example, after building a new validation set* to test performance offline, I tried:</p>
<ul>
<li>Recommending users who follow you (but you don’t follow in return)</li>
<li>Recommending users similar to you (when representing users as sets of their followers, and using cosine similarity and Jaccard similarity as the similarity metric)</li>
<li>Recommending users based on a personalized PageRank score</li>
<li>Recommending users that the people you follow also follow</li>
</ul>
<p>And so on, combining the votes of these algorithms in a fairly ad-hoc way (e.g., by taking the majority vote or by ordering by the number of followers).</p>
<p>This worked quite well actually, but I’d been planning to move on to a more machine learned model-based approach from the beginning, so I did that next.</p>
<p>*My validation set was formed by deleting random edges from the full training set. A slightly better approach, as mentioned above, might have been to more accurately simulate the distribution of the official test set, but I didn’t end up trying this out myself.</p>
<h1>Candidate Selection</h1>
<p>In order to run a machine learning algorithm to recommend edges (which would take two nodes, a source and a candidate destination, and generate a score measuring the likelihood that the source would follow the destination), it’s necessary to prune the set of candidates to run the algorithm on.</p>
<p>I used two approaches for this filtering step, both based on random walks on the graph.</p>
<h2>Personalized PageRank</h2>
<p>The first approach was to calculate a personalized PageRank around each source node.</p>
<p>Briefly, a personalized PageRank is like standard PageRank, except that when randomly teleporting to a new node, the surfer always teleports back to the given source node being personalized (rather than to a node chosen uniformly at random, as in the classic PageRank algorithm).</p>
<p>That is, the random surfer in the personalized PageRank model works as follows:</p>
<ul>
<li>He starts at the source node $X$ that we want to calculate a personalized PageRank around.</li>
<li>At step $i$: with probability $p$, the surfer moves to a neighboring node chosen uniformly at random; with probability $1-p$, the surfer instead teleports back to the original source node $X$.</li>
<li>The limiting probability that the surfer is at node $N$ is then the personalized PageRank score of node $N$ around $X$.</li>
</ul>
<p>Here’s some Scala code that computes approximate personalized PageRank scores and takes the highest-scoring nodes as the candidates to feed into the machine learning model:</p>
<figcaption><span>Personalized PageRank</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
<span class="line-number">19</span>
<span class="line-number">20</span>
<span class="line-number">21</span>
<span class="line-number">22</span>
<span class="line-number">23</span>
<span class="line-number">24</span>
<span class="line-number">25</span>
<span class="line-number">26</span>
<span class="line-number">27</span>
<span class="line-number">28</span>
<span class="line-number">29</span>
<span class="line-number">30</span>
<span class="line-number">31</span>
<span class="line-number">32</span>
<span class="line-number">33</span>
<span class="line-number">34</span>
<span class="line-number">35</span>
<span class="line-number">36</span>
<span class="line-number">37</span>
<span class="line-number">38</span>
<span class="line-number">39</span>
<span class="line-number">40</span>
<span class="line-number">41</span>
<span class="line-number">42</span>
<span class="line-number">43</span>
<span class="line-number">44</span>
<span class="line-number">45</span>
<span class="line-number">46</span>
<span class="line-number">47</span>
<span class="line-number">48</span>
<span class="line-number">49</span>
<span class="line-number">50</span>
<span class="line-number">51</span>
<span class="line-number">52</span>
<span class="line-number">53</span>
<span class="line-number">54</span>
<span class="line-number">55</span>
<span class="line-number">56</span>
<span class="line-number">57</span>
<span class="line-number">58</span>
<span class="line-number">59</span>
<span class="line-number">60</span>
<span class="line-number">61</span>
<span class="line-number">62</span>
<span class="line-number">63</span>
<span class="line-number">64</span>
<span class="line-number">65</span>
<span class="line-number">66</span>
<span class="line-number">67</span>
</pre></td><td class="code"><pre><code class="scala"><span class="line"><span class="cm">/**</span>
</span><span class="line"><span class="cm"> * Calculate a personalized PageRank around the given user, and return </span>
</span><span class="line"><span class="cm"> * a list of the nodes with the highest personalized PageRank scores.</span>
</span><span class="line"><span class="cm"> *</span>
</span><span class="line"><span class="cm"> * @return A list of (node, probability of landing at this node after</span>
</span><span class="line"><span class="cm"> * running a personalized PageRank for K iterations) pairs.</span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"><span class="k">def</span> <span class="n">pageRank</span><span class="o">(</span><span class="n">user</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">Int</span>, <span class="kt">Double</span><span class="o">)]</span> <span class="k">=</span> <span class="o">{</span>
</span><span class="line"> <span class="c1">// This map holds the probability of landing at each node, up to the </span>
</span><span class="line"> <span class="c1">// current iteration.</span>
</span><span class="line"> <span class="k">val</span> <span class="n">probs</span> <span class="k">=</span> <span class="nc">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Double</span><span class="o">]()</span>
</span><span class="line"> <span class="n">probs</span><span class="o">(</span><span class="n">user</span><span class="o">)</span> <span class="k">=</span> <span class="mi">1</span> <span class="c1">// We start at this user.</span>
</span><span class="line">
</span><span class="line"> <span class="k">val</span> <span class="n">pageRankProbs</span> <span class="k">=</span> <span class="n">pageRankHelper</span><span class="o">(</span><span class="n">start</span><span class="o">,</span> <span class="n">probs</span><span class="o">,</span> <span class="nc">NumPagerankIterations</span><span class="o">)</span>
</span><span class="line"> <span class="n">pageRankProbs</span><span class="o">.</span><span class="n">toList</span>
</span><span class="line"> <span class="o">.</span><span class="n">sortBy</span> <span class="o">{</span> <span class="o">-</span><span class="k">_</span><span class="o">.</span><span class="n">_2</span> <span class="o">}</span>
</span><span class="line"> <span class="o">.</span><span class="n">filter</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">node</span><span class="o">,</span> <span class="n">score</span><span class="o">)</span> <span class="k">=></span>
</span><span class="line"> <span class="o">!</span><span class="n">getFollowings</span><span class="o">(</span><span class="n">user</span><span class="o">).</span><span class="n">contains</span><span class="o">(</span><span class="n">node</span><span class="o">)</span> <span class="o">&&</span> <span class="n">node</span> <span class="o">!=</span> <span class="n">user</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line"> <span class="o">.</span><span class="n">take</span><span class="o">(</span><span class="nc">MaxNodesToKeep</span><span class="o">)</span>
</span><span class="line"><span class="o">}</span>
</span><span class="line">
</span><span class="line"><span class="cm">/**</span>
</span><span class="line"><span class="cm"> * Simulates running a personalized PageRank for one iteration.</span>
</span><span class="line"><span class="cm"> *</span>
</span><span class="line"><span class="cm"> * Parameters:</span>
</span><span class="line"><span class="cm"> * start - the start node to calculate the personalized PageRank around</span>
</span><span class="line"><span class="cm"> * probs - a map from nodes to the probability of being at that node at </span>
</span><span class="line"><span class="cm"> * the start of the current iteration</span>
</span><span class="line"><span class="cm"> * numIterations - the number of iterations remaining</span>
</span><span class="line"><span class="cm"> * alpha - with probability alpha, we follow a neighbor; with probability</span>
</span><span class="line"><span class="cm"> * 1 - alpha, we teleport back to the start node</span>
</span><span class="line"><span class="cm"> *</span>
</span><span class="line"><span class="cm"> * @return A map of node -> probability of landing at that node after the</span>
</span><span class="line"><span class="cm"> * specified number of iterations.</span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"><span class="k">def</span> <span class="n">pageRankHelper</span><span class="o">(</span><span class="n">start</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">probs</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Double</span><span class="o">],</span> <span class="n">numIterations</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span>
</span><span class="line"> <span class="n">alpha</span><span class="k">:</span> <span class="kt">Double</span> <span class="o">=</span> <span class="mf">0.5</span><span class="o">)</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Double</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
</span><span class="line"> <span class="k">if</span> <span class="o">(</span><span class="n">numIterations</span> <span class="o"><=</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
</span><span class="line"> <span class="n">probs</span>
</span><span class="line"> <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
</span><span class="line"> <span class="c1">// Holds the updated set of probabilities, after this iteration.</span>
</span><span class="line"> <span class="k">val</span> <span class="n">probsPropagated</span> <span class="k">=</span> <span class="nc">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Double</span><span class="o">]()</span>
</span><span class="line">
</span><span class="line"> <span class="c1">// With probability 1 - alpha, we teleport back to the start node.</span>
</span><span class="line"> <span class="n">probsPropagated</span><span class="o">(</span><span class="n">start</span><span class="o">)</span> <span class="k">=</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">alpha</span>
</span><span class="line">
</span><span class="line"> <span class="c1">// Propagate the previous probabilities...</span>
</span><span class="line"> <span class="n">probs</span><span class="o">.</span><span class="n">foreach</span> <span class="o">{</span> <span class="k">case</span> <span class="o">(</span><span class="n">node</span><span class="o">,</span> <span class="n">prob</span><span class="o">)</span> <span class="k">=></span>
</span><span class="line"> <span class="k">val</span> <span class="n">forwards</span> <span class="k">=</span> <span class="n">getFollowings</span><span class="o">(</span><span class="n">node</span><span class="o">)</span>
</span><span class="line"> <span class="k">val</span> <span class="n">backwards</span> <span class="k">=</span> <span class="n">getFollowers</span><span class="o">(</span><span class="n">node</span><span class="o">)</span>
</span><span class="line">
</span><span class="line"> <span class="c1">// With probability alpha, we move to a follower...</span>
</span><span class="line"> <span class="c1">// And each node distributes its current probability equally to </span>
</span><span class="line"> <span class="c1">// its neighbors.</span>
</span><span class="line"> <span class="k">val</span> <span class="n">probToPropagate</span> <span class="k">=</span> <span class="n">alpha</span> <span class="o">*</span> <span class="n">prob</span> <span class="o">/</span> <span class="o">(</span><span class="n">forwards</span><span class="o">.</span><span class="n">size</span> <span class="o">+</span> <span class="n">backwards</span><span class="o">.</span><span class="n">size</span><span class="o">)</span>
</span><span class="line"> <span class="o">(</span><span class="n">forwards</span><span class="o">.</span><span class="n">toList</span> <span class="o">++</span> <span class="n">backwards</span><span class="o">.</span><span class="n">toList</span><span class="o">).</span><span class="n">foreach</span> <span class="o">{</span> <span class="n">neighbor</span> <span class="k">=></span>
</span><span class="line"> <span class="k">if</span> <span class="o">(!</span><span class="n">probsPropagated</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">neighbor</span><span class="o">))</span> <span class="o">{</span>
</span><span class="line"> <span class="n">probsPropagated</span><span class="o">(</span><span class="n">neighbor</span><span class="o">)</span> <span class="k">=</span> <span class="mi">0</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line"> <span class="n">probsPropagated</span><span class="o">(</span><span class="n">neighbor</span><span class="o">)</span> <span class="o">+=</span> <span class="n">probToPropagate</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line">
</span><span class="line"> <span class="n">pageRankHelper</span><span class="o">(</span><span class="n">start</span><span class="o">,</span> <span class="n">probsPropagated</span><span class="o">,</span> <span class="n">numIterations</span> <span class="o">-</span> <span class="mi">1</span><span class="o">,</span> <span class="n">alpha</span><span class="o">)</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line"><span class="o">}</span>
</span></code></pre></td></tr></table></div>
<h2>Propagation Score</h2>
<p>Another approach I used, based on <a href="http://www.kaggle.com/c/FacebookRecruiting/forums/t/2082/0-711-is-the-new-0">a proposal by another contestant on the Kaggle forums</a>, works as follows:</p>
<ul>
<li>Start at a specified user node and give it some score.</li>
<li>In the first iteration, this user propagates its score equally to its neighbors.</li>
<li>In the second iteration, each user duplicates and keeps half of its score S. It then propagates S equally to its neighbors.</li>
<li>In subsequent iterations, the process is repeated, except that neighbors reached via a backwards link don’t duplicate and keep half of their score. (The idea is that we want the score to reach followees and not followers.)</li>
</ul>
<p>Here’s some Scala code to calculate these propagation scores:</p>
<figcaption><span>Propagation Score</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
<span class="line-number">19</span>
<span class="line-number">20</span>
<span class="line-number">21</span>
<span class="line-number">22</span>
<span class="line-number">23</span>
<span class="line-number">24</span>
<span class="line-number">25</span>
<span class="line-number">26</span>
<span class="line-number">27</span>
<span class="line-number">28</span>
<span class="line-number">29</span>
<span class="line-number">30</span>
<span class="line-number">31</span>
<span class="line-number">32</span>
<span class="line-number">33</span>
<span class="line-number">34</span>
<span class="line-number">35</span>
<span class="line-number">36</span>
<span class="line-number">37</span>
<span class="line-number">38</span>
<span class="line-number">39</span>
<span class="line-number">40</span>
<span class="line-number">41</span>
<span class="line-number">42</span>
<span class="line-number">43</span>
<span class="line-number">44</span>
<span class="line-number">45</span>
<span class="line-number">46</span>
<span class="line-number">47</span>
<span class="line-number">48</span>
<span class="line-number">49</span>
<span class="line-number">50</span>
<span class="line-number">51</span>
<span class="line-number">52</span>
<span class="line-number">53</span>
<span class="line-number">54</span>
<span class="line-number">55</span>
<span class="line-number">56</span>
<span class="line-number">57</span>
<span class="line-number">58</span>
<span class="line-number">59</span>
<span class="line-number">60</span>
<span class="line-number">61</span>
</pre></td><td class="code"><pre><code class="scala"><span class="line"><span class="cm">/**</span>
</span><span class="line"><span class="cm"> * Calculate propagation scores around the current user.</span>
</span><span class="line"><span class="cm"> *</span>
</span><span class="line"><span class="cm"> * In the first propagation round, we</span>
</span><span class="line"><span class="cm"> *</span>
</span><span class="line"><span class="cm"> * - Give the starting node N an initial score S.</span>
</span><span class="line"><span class="cm"> * - Propagate the score equally to each of N's neighbors (followers </span>
</span><span class="line"><span class="cm"> * and followings).</span>
</span><span class="line"><span class="cm"> * - Each first-level neighbor then duplicates and keeps half of its score</span>
</span><span class="line"><span class="cm"> * and then propagates the original again to its neighbors.</span>
</span><span class="line"><span class="cm"> *</span>
</span><span class="line"><span class="cm"> * In further rounds, neighbors then repeat the process, except that neighbors </span>
</span><span class="line"><span class="cm"> * traveled to via a backwards/follower link don't keep half of their score.</span>
</span><span class="line"><span class="cm"> *</span>
</span><span class="line"><span class="cm"> * @return a sorted list of (node, propagation score) pairs.</span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"><span class="k">def</span> <span class="n">propagate</span><span class="o">(</span><span class="n">user</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">List</span><span class="o">[(</span><span class="kt">Int</span>, <span class="kt">Double</span><span class="o">)]</span> <span class="k">=</span> <span class="o">{</span>
</span><span class="line"> <span class="k">val</span> <span class="n">scores</span> <span class="k">=</span> <span class="nc">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Double</span><span class="o">]()</span>
</span><span class="line">
</span><span class="line"> <span class="c1">// We propagate the score equally to all neighbors.</span>
</span><span class="line"> <span class="k">val</span> <span class="n">scoreToPropagate</span> <span class="k">=</span> <span class="mf">1.0</span> <span class="o">/</span> <span class="o">(</span><span class="n">getFollowings</span><span class="o">(</span><span class="n">user</span><span class="o">).</span><span class="n">size</span> <span class="o">+</span> <span class="n">getFollowers</span><span class="o">(</span><span class="n">user</span><span class="o">).</span><span class="n">size</span><span class="o">)</span>
</span><span class="line">
</span><span class="line"> <span class="o">(</span><span class="n">getFollowings</span><span class="o">(</span><span class="n">user</span><span class="o">).</span><span class="n">toList</span> <span class="o">++</span> <span class="n">getFollowers</span><span class="o">(</span><span class="n">user</span><span class="o">).</span><span class="n">toList</span><span class="o">).</span><span class="n">foreach</span> <span class="o">{</span> <span class="n">x</span> <span class="k">=></span>
</span><span class="line"> <span class="c1">// Propagate the score...</span>
</span><span class="line"> <span class="n">continuePropagation</span><span class="o">(</span><span class="n">scores</span><span class="o">,</span> <span class="n">x</span><span class="o">,</span> <span class="n">scoreToPropagate</span><span class="o">,</span> <span class="mi">1</span><span class="o">)</span>
</span><span class="line"> <span class="c1">// ...and make sure it keeps half of it for itself.</span>
</span><span class="line"> <span class="n">scores</span><span class="o">(</span><span class="n">x</span><span class="o">)</span> <span class="k">=</span> <span class="n">scores</span><span class="o">.</span><span class="n">getOrElse</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="mi">0</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span> <span class="o">+</span> <span class="n">scoreToPropagate</span> <span class="o">/</span> <span class="mi">2</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line">
</span><span class="line"> <span class="n">scores</span><span class="o">.</span><span class="n">toList</span><span class="o">.</span><span class="n">sortBy</span> <span class="o">{</span> <span class="o">-</span><span class="k">_</span><span class="o">.</span><span class="n">_2</span> <span class="o">}</span>
</span><span class="line"> <span class="o">.</span><span class="n">filter</span> <span class="o">{</span> <span class="n">nodeAndScore</span> <span class="k">=></span>
</span><span class="line"> <span class="k">val</span> <span class="n">node</span> <span class="k">=</span> <span class="n">nodeAndScore</span><span class="o">.</span><span class="n">_1</span>
</span><span class="line"> <span class="o">!</span><span class="n">getFollowings</span><span class="o">(</span><span class="n">user</span><span class="o">).</span><span class="n">contains</span><span class="o">(</span><span class="n">node</span><span class="o">)</span> <span class="o">&&</span> <span class="n">node</span> <span class="o">!=</span> <span class="n">user</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line"> <span class="o">.</span><span class="n">take</span><span class="o">(</span><span class="nc">MaxNodesToKeep</span><span class="o">)</span>
</span><span class="line"><span class="o">}</span>
</span><span class="line">
</span><span class="line"><span class="cm">/**</span>
</span><span class="line"><span class="cm"> * In further rounds, neighbors repeat the process above, except that neighbors</span>
</span><span class="line"><span class="cm"> * traveled to via a backwards/follower link don't keep half of their score.</span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"><span class="k">def</span> <span class="n">continuePropagation</span><span class="o">(</span><span class="n">scores</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Double</span><span class="o">],</span> <span class="n">user</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">score</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span>
</span><span class="line"> <span class="n">currIteration</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span>
</span><span class="line"> <span class="k">if</span> <span class="o">(</span><span class="n">currIteration</span> <span class="o"><</span> <span class="nc">NumIterations</span> <span class="o">&&</span> <span class="n">score</span> <span class="o">></span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
</span><span class="line"> <span class="k">val</span> <span class="n">scoreToPropagate</span> <span class="k">=</span> <span class="n">score</span> <span class="o">/</span> <span class="o">(</span><span class="n">getFollowings</span><span class="o">(</span><span class="n">user</span><span class="o">).</span><span class="n">size</span> <span class="o">+</span> <span class="n">getFollowers</span><span class="o">(</span><span class="n">user</span><span class="o">).</span><span class="n">size</span><span class="o">)</span>
</span><span class="line">
</span><span class="line"> <span class="n">getFollowings</span><span class="o">(</span><span class="n">user</span><span class="o">).</span><span class="n">foreach</span> <span class="o">{</span> <span class="n">x</span> <span class="k">=></span>
</span><span class="line"> <span class="c1">// Propagate the score... </span>
</span><span class="line"> <span class="n">continuePropagation</span><span class="o">(</span><span class="n">scores</span><span class="o">,</span> <span class="n">x</span><span class="o">,</span> <span class="n">scoreToPropagate</span><span class="o">,</span> <span class="n">currIteration</span> <span class="o">+</span> <span class="mi">1</span><span class="o">)</span>
</span><span class="line"> <span class="c1">// ...and make sure it keeps half of it for itself. </span>
</span><span class="line"> <span class="n">scores</span><span class="o">(</span><span class="n">x</span><span class="o">)</span> <span class="k">=</span> <span class="n">scores</span><span class="o">.</span><span class="n">getOrElse</span><span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="mi">0</span><span class="k">:</span> <span class="kt">Double</span><span class="o">)</span> <span class="o">+</span> <span class="n">scoreToPropagate</span> <span class="o">/</span> <span class="mi">2</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line">
</span><span class="line"> <span class="n">getFollowers</span><span class="o">(</span><span class="n">user</span><span class="o">).</span><span class="n">foreach</span> <span class="o">{</span> <span class="n">x</span> <span class="k">=></span>
</span><span class="line"> <span class="c1">// Propagate the score...</span>
</span><span class="line"> <span class="n">continuePropagation</span><span class="o">(</span><span class="n">scores</span><span class="o">,</span> <span class="n">x</span><span class="o">,</span> <span class="n">scoreToPropagate</span><span class="o">,</span> <span class="n">currIteration</span> <span class="o">+</span> <span class="mi">1</span><span class="o">)</span>
</span><span class="line"> <span class="c1">// ...but backward links (except for the starting node's immediate</span>
</span><span class="line"> <span class="c1">// neighbors) don't keep any score for themselves.</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line"><span class="o">}</span>
</span></code></pre></td></tr></table></div>
<p>I played around with tweaking some parameters in both approaches (e.g., weighting followers and followees differently), but the natural defaults (as used in the code above) ended up performing the best.</p>
<h1>Features</h1>
<p>After pruning the set of candidate destination nodes to a more feasible level, I fed pairs of (source, destination) nodes into a machine learning model. From each pair, I extracted around 30 features in total.</p>
<p>As mentioned above, one feature that worked quite well on its own was whether the destination node already follows the source.</p>
<p>I also used a wide set of similarity-based features, for example, the Jaccard similarity between the source and destination when both are represented as sets of their followers, when both are represented as sets of their followees, or when one is represented as a set of followers while the other is represented as a set of followees.</p>
<figcaption><span>Similarity Metrics</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
<span class="line-number">19</span>
<span class="line-number">20</span>
<span class="line-number">21</span>
<span class="line-number">22</span>
<span class="line-number">23</span>
<span class="line-number">24</span>
<span class="line-number">25</span>
<span class="line-number">26</span>
<span class="line-number">27</span>
<span class="line-number">28</span>
<span class="line-number">29</span>
<span class="line-number">30</span>
<span class="line-number">31</span>
<span class="line-number">32</span>
<span class="line-number">33</span>
<span class="line-number">34</span>
<span class="line-number">35</span>
<span class="line-number">36</span>
<span class="line-number">37</span>
<span class="line-number">38</span>
<span class="line-number">39</span>
<span class="line-number">40</span>
<span class="line-number">41</span>
<span class="line-number">42</span>
<span class="line-number">43</span>
<span class="line-number">44</span>
<span class="line-number">45</span>
<span class="line-number">46</span>
<span class="line-number">47</span>
<span class="line-number">48</span>
<span class="line-number">49</span>
</pre></td><td class="code"><pre><code class="scala"><span class="line"><span class="k">abstract</span> <span class="k">class</span> <span class="nc">SimilarityMetric</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="o">{</span>
</span><span class="line"> <span class="k">def</span> <span class="n">apply</span><span class="o">(</span><span class="n">set1</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">T</span><span class="o">],</span> <span class="n">set2</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">T</span><span class="o">])</span><span class="k">:</span> <span class="kt">Double</span><span class="o">;</span>
</span><span class="line"><span class="o">}</span>
</span><span class="line">
</span><span class="line"><span class="k">object</span> <span class="nc">JaccardSimilarity</span> <span class="k">extends</span> <span class="nc">SimilarityMetric</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="o">{</span>
</span><span class="line"> <span class="cm">/**</span>
</span><span class="line"><span class="cm"> * Returns the Jaccard similarity between two sets, 0 if both are empty.</span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"> <span class="k">def</span> <span class="n">apply</span><span class="o">(</span><span class="n">set1</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">Int</span><span class="o">],</span> <span class="n">set2</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Double</span> <span class="o">=</span> <span class="o">{</span>
</span><span class="line"> <span class="k">val</span> <span class="n">union</span> <span class="k">=</span> <span class="o">(</span><span class="n">set1</span><span class="o">.</span><span class="n">union</span><span class="o">(</span><span class="n">set2</span><span class="o">)).</span><span class="n">size</span>
</span><span class="line">
</span><span class="line"> <span class="k">if</span> <span class="o">(</span><span class="n">union</span> <span class="o">==</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
</span><span class="line"> <span class="mi">0</span>
</span><span class="line"> <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
</span><span class="line"> <span class="o">(</span><span class="n">set1</span> <span class="o">&</span> <span class="n">set2</span><span class="o">).</span><span class="n">size</span><span class="o">.</span><span class="n">toFloat</span> <span class="o">/</span> <span class="n">union</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line">
</span><span class="line"><span class="o">}</span>
</span><span class="line">
</span><span class="line"><span class="k">object</span> <span class="nc">CosineSimilarity</span> <span class="k">extends</span> <span class="nc">SimilarityMetric</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="o">{</span>
</span><span class="line"> <span class="cm">/**</span>
</span><span class="line"><span class="cm"> * Returns the cosine similarity between two sets, 0 if both are empty.</span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"> <span class="k">def</span> <span class="n">apply</span><span class="o">(</span><span class="n">set1</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">Int</span><span class="o">],</span> <span class="n">set2</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Double</span> <span class="o">=</span> <span class="o">{</span>
</span><span class="line"> <span class="k">if</span> <span class="o">(</span><span class="n">set1</span><span class="o">.</span><span class="n">size</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">set2</span><span class="o">.</span><span class="n">size</span> <span class="o">==</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
</span><span class="line"> <span class="mi">0</span>
</span><span class="line"> <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
</span><span class="line"> <span class="o">(</span><span class="n">set1</span> <span class="o">&</span> <span class="n">set2</span><span class="o">).</span><span class="n">size</span><span class="o">.</span><span class="n">toFloat</span> <span class="o">/</span> <span class="o">(</span><span class="n">math</span><span class="o">.</span><span class="n">sqrt</span><span class="o">(</span><span class="n">set1</span><span class="o">.</span><span class="n">size</span> <span class="o">*</span> <span class="n">set2</span><span class="o">.</span><span class="n">size</span><span class="o">))</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line">
</span><span class="line"><span class="o">}</span>
</span><span class="line">
</span><span class="line"><span class="c1">// ************</span>
</span><span class="line"><span class="c1">// * FEATURES *</span>
</span><span class="line"><span class="c1">// ************</span>
</span><span class="line">
</span><span class="line"><span class="cm">/**</span>
</span><span class="line"><span class="cm"> * Returns the similarity between user1 and user2 when both are represented as</span>
</span><span class="line"><span class="cm"> * sets of followers.</span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"><span class="k">def</span> <span class="n">similarityByFollowers</span><span class="o">(</span><span class="n">user1</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">user2</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span>
</span><span class="line"> <span class="o">(</span><span class="k">implicit</span> <span class="n">similarity</span><span class="k">:</span> <span class="kt">SimilarityMetric</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Double</span> <span class="o">=</span> <span class="o">{</span>
</span><span class="line"> <span class="n">similarity</span><span class="o">.</span><span class="n">apply</span><span class="o">(</span><span class="n">getFollowersWithout</span><span class="o">(</span><span class="n">user1</span><span class="o">,</span> <span class="n">user2</span><span class="o">),</span>
</span><span class="line"> <span class="n">getFollowersWithout</span><span class="o">(</span><span class="n">user2</span><span class="o">,</span> <span class="n">user1</span><span class="o">))</span>
</span><span class="line"><span class="o">}</span>
</span><span class="line">
</span><span class="line"><span class="c1">// etc.</span>
</span></code></pre></td></tr></table></div>
<p>Along the same lines, I also computed a similarity score between the destination node and the source node’s followees, and several variations thereof.</p>
<figcaption><span>Extended Similarity Scores</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
</pre></td><td class="code"><pre><code class="scala"><span class="line"><span class="cm">/**</span>
</span><span class="line"><span class="cm"> * Iterate over each of user1's followings, compute their similarity with</span>
</span><span class="line"><span class="cm"> * user2 when both are represented as sets of followers, and return the </span>
</span><span class="line"><span class="cm"> * sum of these similarities.</span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"><span class="k">def</span> <span class="n">followerBasedSimilarityToFollowing</span><span class="o">(</span><span class="n">user1</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">user2</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span>
</span><span class="line"> <span class="o">(</span><span class="k">implicit</span> <span class="n">similarity</span><span class="k">:</span> <span class="kt">SimilarityMetric</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Double</span> <span class="o">=</span> <span class="o">{</span>
</span><span class="line"> <span class="n">getFollowingsWithout</span><span class="o">(</span><span class="n">user1</span><span class="o">,</span> <span class="n">user2</span><span class="o">)</span>
</span><span class="line"> <span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">similarityByFollowers</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="n">user2</span><span class="o">)(</span><span class="n">similarity</span><span class="o">)</span> <span class="o">}</span>
</span><span class="line"> <span class="o">.</span><span class="n">sum</span>
</span><span class="line"><span class="o">}</span>
</span></code></pre></td></tr></table></div>
<p>Other features included the number of followers and followees of each node, the ratio of these, the personalized PageRank and propagation scores themselves, the number of followers in common, and triangle/closure-type features (e.g., whether the source node is friends with a node X who in turn is a friend of the destination node).</p>
<p>If I had had more time, I would probably have tried weighted and more regularized versions of some of these features as well (e.g., downweighting nodes with large numbers of followers when computing cosine similarity scores based on followees, or shrinking the scores of nodes we have little information about).</p>
<h1>Feature Understanding</h1>
<p>But what are these features actually <em>doing</em>? Let’s use the same app I built before to take a look.</p>
<p>Here’s the local network of node 317 (different from the node above), where each node is colored by its personalized PageRank (higher scores are in darker red):</p>
<p><a href="http://i.imgur.com/0OpjUAK.png"><img src="http://i.imgur.com/0OpjUAK.png" alt="317 - Personalized PageRank" /></a></p>
<p>If we look at the following vs. follower relationships of the central node (recall that purple is friends, teal is followings, orange is followers):</p>
<p><a href="http://i.imgur.com/IPWrww9.png"><img src="http://i.imgur.com/IPWrww9.png" alt="317 - Personalized PageRank" /></a></p>
<p>…we can see that, as expected (because edges that represented both following and follower were double-weighted in my PageRank calculation), the darkest red nodes are those that are friends with the central node, while those in a following-only or follower-only relationship have a lower score.</p>
<p>How does the propagation score compare to personalized PageRank? Here, I colored each node according to the log ratio of its propagation score and personalized PageRank:</p>
<p><a href="http://i.imgur.com/aenZaq6.png"><img src="http://i.imgur.com/aenZaq6.png" alt="317 - Log Ratio" /></a></p>
<p>Comparing this coloring with the local follow/follower network:</p>
<p><a href="http://i.imgur.com/n6AZWyq.png"><img src="http://i.imgur.com/n6AZWyq.png" alt="317 - Local Network of Node" /></a></p>
<p>…we can see that followed nodes (in teal) receive a higher propagation weight than friend nodes (in purple), while follower nodes (in orange) receive almost no propagation score at all.</p>
<p>Going back to node 1, let’s look at a different metric. Here, each node is colored according to its Jaccard similarity with the source, when nodes are represented by the set of their followers:</p>
<p><a href="http://i.imgur.com/hVnnFAO.png"><img src="http://i.imgur.com/hVnnFAO.png" alt="1 - Similarity by Followers" /></a></p>
<p>We can see that, while the PageRank and propagation metrics tended to favor nodes <em>close</em> to the central node, the Jaccard similarity feature helps us explore nodes that are further out.</p>
<p>However, if we look the high-scoring nodes more closely, we see that they often have only a single connection to the rest of the network:</p>
<p><a href="http://i.imgur.com/mRArIFf.png"><img src="http://i.imgur.com/mRArIFf.png" alt="1 - Single Connection" /></a></p>
<p>In other words, their high Jaccard similarity is due to the fact that they don’t have many connections to begin with. This suggests that some regularization or shrinking is in order.</p>
<p>So here’s a regularized version of Jaccard similarity, where we downweight nodes with few connections:</p>
<p><a href="http://i.imgur.com/1fZWX45.png"><img src="http://i.imgur.com/1fZWX45.png" alt="1 - Regularized" /></a></p>
<p>We can see that the outlier nodes are much more muted this time around.</p>
<p>For a starker difference, compare the following two graphs of the Jaccard similarity metric around node 317 (the first graph is an unregularized version, the second is regularized):</p>
<p><a href="http://i.imgur.com/a7QWscH.png"><img src="http://i.imgur.com/a7QWscH.png" alt="317 - Unregularized" /></a></p>
<p><a href="http://i.imgur.com/MPwk5Bv.png"><img src="http://i.imgur.com/MPwk5Bv.png" alt="317 - Regularized" /></a></p>
<p>Notice, in particular, how the popular node in the top left and the popular nodes at the bottom have a much higher score when we regularize.</p>
<p>And again, there are other networks and features I haven’t mentioned here, so play around and discover them on the <a href="http://link-prediction.herokuapp.com/">app</a> itself.</p>
<h1>Models</h1>
<p>For the machine learning algorithms on top of my features, I experimented with two types of models: logistic regression (using both L1 and L2 regularization) and random forests. (If I had more time, I would probably have done some more parameter tuning and maybe tried gradient boosted trees as well.)</p>
<p>So what is a random forest? I wrote an <a href="http://www.quora.com/Random-Forests/How-do-random-forests-work-in-laymans-terms/answer/Edwin-Chen-1">old (layman’s) post</a> on it <a href="http://www.quora.com/Random-Forests/How-do-random-forests-work-in-laymans-terms/answer/Edwin-Chen-1">here</a>, but since nobody ever clicks on these links, let’s copy it over:</p>
<blockquote><p>Suppose you’re very indecisive, so whenever you want to watch a movie, you ask your friend Willow if she thinks you’ll like it. In order to answer, Willow first needs to figure out what movies you like, so you give her a bunch of movies and tell her whether you liked each one or not (i.e., you give her a labeled training set). Then, when you ask her if she thinks you’ll like movie X or not, she plays a 20 questions-like game with IMDB, asking questions like “Is X a romantic movie?”, “Does Johnny Depp star in X?”, and so on. She asks more informative questions first (i.e., she maximizes the information gain of each question), and gives you a yes/no answer at the end.</p><p> Thus, Willow is a decision tree for your movie preferences.</p><p> But Willow is only human, so she doesn’t always generalize your preferences very well (i.e., she overfits). In order to get more accurate recommendations, you’d like to ask a bunch of your friends, and watch movie X if most of them say they think you’ll like it. That is, instead of asking only Willow, you want to ask Woody, Apple, and Cartman as well, and they vote on whether you’ll like a movie (i.e., you build an ensemble classifier, aka a forest in this case).</p><p> Now you don’t want each of your friends to do the same thing and give you the same answer, so you first give each of them slightly different data. After all, you’re not absolutely sure of your preferences yourself – you told Willow you loved Titanic, but maybe you were just happy that day because it was your birthday, so maybe some of your friends shouldn’t use the fact that you liked Titanic in making their recommendations. Or maybe you told her you loved Cinderella, but actually you *really really* loved it, so some of your friends should give Cinderella more weight. So instead of giving your friends the same data you gave Willow, you give them slightly perturbed versions. You don’t change your love/hate decisions, you just say you love/hate some movies a little more or less (you give each of your friends a bootstrapped version of your original training data). For example, whereas you told Willow that you liked Black Swan and Harry Potter and disliked Avatar, you tell Woody that you liked Black Swan so much you watched it twice, you disliked Avatar, and don’t mention Harry Potter at all.</p><p> By using this ensemble, you hope that while each of your friends gives somewhat idiosyncratic recommendations (Willow thinks you like vampire movies more than you do, Woody thinks you like Pixar movies, and Cartman thinks you just hate everything), the errors get canceled out in the majority. Thus, your friends now form a bagged (bootstrap aggregated) forest of your movie preferences.</p><p> There’s still one problem with your data, however. While you loved both Titanic and Inception, it wasn’t because you like movies that star Leonardio DiCaprio. Maybe you liked both movies for other reasons. Thus, you don’t want your friends to all base their recommendations on whether Leo is in a movie or not. So when each friend asks IMDB a question, only a random subset of the possible questions is allowed (i.e., when you’re building a decision tree, at each node you use some randomness in selecting the attribute to split on, say by randomly selecting an attribute or by selecting an attribute from a random subset). This means your friends aren’t allowed to ask whether Leonardo DiCaprio is in the movie whenever they want. So whereas previously you injected randomness at the data level, by perturbing your movie preferences slightly, now you’re injecting randomness at the model level, by making your friends ask different questions at different times.</p><p> And so your friends now form a random forest.</p></blockquote>
<p>Moving on, I essentially trained <a href="http://scikit-learn.org/stable/">scikit-learn</a>’s classifiers on an equal split of true and false edges (sampled from the output of my pruning step, in order to match the distribution I’d get when applying my algorithm to the official test set), and compared performance on the validation set I made, with a small amount of parameter tuning:</p>
<figcaption><span>Random Forest</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
</pre></td><td class="code"><pre><code class="python"><span class="line"><span class="c">########################################</span>
</span><span class="line"><span class="c"># STEP 1: Read in the training examples.</span>
</span><span class="line"><span class="c">########################################</span>
</span><span class="line"><span class="n">truths</span> <span class="o">=</span> <span class="p">[]</span> <span class="c"># A truth is 1 (for a known true edge) or 0 (for a false edge).</span>
</span><span class="line"><span class="n">training_examples</span> <span class="o">=</span> <span class="p">[]</span> <span class="c"># Each training example is an array of features.</span>
</span><span class="line"><span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="nb">open</span><span class="p">(</span><span class="n">TRAINING_SET_WITH_FEATURES_FILENAME</span><span class="p">):</span>
</span><span class="line"> <span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="nb">float</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">","</span><span class="p">)]</span>
</span><span class="line"> <span class="n">truth</span> <span class="o">=</span> <span class="n">values</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span class="line"> <span class="n">training_example_features</span> <span class="o">=</span> <span class="n">values</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span>
</span><span class="line">
</span><span class="line"> <span class="n">truths</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">truth</span><span class="p">)</span>
</span><span class="line"> <span class="n">training_examples</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">training_example_features</span><span class="p">)</span>
</span><span class="line">
</span><span class="line"><span class="c">#############################</span>
</span><span class="line"><span class="c"># STEP 2: Train a classifier.</span>
</span><span class="line"><span class="c">#############################</span>
</span><span class="line"><span class="n">rf</span> <span class="o">=</span> <span class="n">RandomForestClassifier</span><span class="p">(</span><span class="n">n_estimators</span> <span class="o">=</span> <span class="mi">500</span><span class="p">,</span> <span class="n">compute_importances</span> <span class="o">=</span> <span class="bp">True</span><span class="p">,</span> <span class="n">oob_score</span> <span class="o">=</span> <span class="bp">True</span><span class="p">)</span>
</span><span class="line"><span class="n">rf</span> <span class="o">=</span> <span class="n">rf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">training_examples</span><span class="p">,</span> <span class="n">truths</span><span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p>So let’s look at the variable importance scores as determined by one of my random forest models, which (unsurprisingly) consistently outperformed logistic regression.</p>
<p><a href="http://i.imgur.com/7D5yi6G.png"><img src="http://i.imgur.com/7D5yi6G.png" alt="Random Forest Importance Scores" /></a></p>
<p>The random forest classifier here is one of my earlier models (using a slightly smaller subset of my full suite of features), where the targeting step consisted of taking the top 25 nodes with the highest propagation scores.</p>
<p>We can see that the most important variables are:</p>
<ul>
<li>Personalized PageRank scores. (I put in both normalized and unnormalized versions, where the normalized versions consisted of taking all the candidates for a particular source node, and scaling them so that the maximum personalized PageRank score was 1.)</li>
<li>Whether the destination node already follows the source.</li>
<li>How similar the source node is to the people the destination node is following, when each node is represented as a set of followers. (Note that this is more or less measuring how likely the destination is to follow the source, which we already saw is a good predictor of whether the source is likely to follow the destination.) Plus several variations on this theme (e.g., how similar the destination node is to the source node’s followers, when each node is represented as a set of followees).</li>
</ul>
<h1>Model Comparison</h1>
<p>How do all of these models compare to each other? Is the random forest model universally better than the logistic regression model, or are there some sets of users for which the logistic regression model actually performs better?</p>
<p>To enable these kinds of comparisons, I made <a href="http://link-prediction.herokuapp.com/comparison">a small module</a> that allows you to select two models and then visualize their sliced performance.</p>
<p><a href="http://link-prediction.herokuapp.com/comparison"><img src="http://i.imgur.com/ahNHp5z.png" alt="PageRank vs. Is Followed By" /></a></p>
<p>(Go ahead, <a href="http://link-prediction.herokuapp.com/comparison">play around</a>.)</p>
<p>Above, I bucketed all test nodes into buckets based on (the logarithm of) their number of followers, and compared the mean average precision of two algorithms: one that recommends nodes to follow using a personalized PageRank alone, and one that recommends nodes that are following the source user but are not followed back in return.</p>
<p>We see that except for the case of 0 followers (where the “is followed by” algorithm can do nothing), the personalized PageRank algorithm gets increasingly better in comparison: at first, the two algorithms have roughly equal performance, but as the source node gets more followers, the personalized PageRank algorithm dominates.</p>
<p>And here’s an embedded version you can interact with directly:</p>
<iframe width="600px" height="500px" src="http://link-prediction.herokuapp.com/comparison?for_embed=true"></iframe>
<p>Admittedly, building a slicer like this is probably overkill for a Kaggle competition, where the set of variables is fairly limited. But imagine having something similar for a real world model, where new algorithms are tried out every week and we can slice the performance by almost any dimension we can imagine (by geography, to make sure we don’t improve Australia at the expense of the UK; by user interests, to see where we could improve the performance of topic inference; by number of user logins, to make sure we don’t sacrifice the performance on new users for the gain of the core).</p>
<h1>Mathematicians do it with Matrices</h1>
<p>Let’s switch directions slightly and think about how we could rewrite our computations in a different, matrix-oriented style. (I didn’t do this in the competition – this is more a preview of another post I’m writing.)</p>
<h2>Personalized PageRank in Scalding</h2>
<p>Personalized PageRank, for example, is an obvious fit for a matrix rewrite. Here’s how it would look in <a href="http://blog.echen.me/2012/02/09/movie-recommendations-and-more-via-mapreduce-and-scalding/">Scalding</a>’s new Matrix library:</p>
<p>(For those who don’t know, Scalding is a Hadoop framework that Twitter released at the beginning of the year; see <a href="http://blog.echen.me/2012/02/09/movie-recommendations-and-more-via-mapreduce-and-scalding/">my post on building a big data recommendation engine in Scalding</a> for an introduction.)</p>
<div class="clearboth"></div>
<figcaption><span>Personalized PageRank, Matrix Style</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
<span class="line-number">19</span>
<span class="line-number">20</span>
<span class="line-number">21</span>
<span class="line-number">22</span>
<span class="line-number">23</span>
<span class="line-number">24</span>
<span class="line-number">25</span>
<span class="line-number">26</span>
<span class="line-number">27</span>
<span class="line-number">28</span>
<span class="line-number">29</span>
<span class="line-number">30</span>
<span class="line-number">31</span>
<span class="line-number">32</span>
<span class="line-number">33</span>
<span class="line-number">34</span>
<span class="line-number">35</span>
<span class="line-number">36</span>
<span class="line-number">37</span>
<span class="line-number">38</span>
<span class="line-number">39</span>
<span class="line-number">40</span>
<span class="line-number">41</span>
<span class="line-number">42</span>
<span class="line-number">43</span>
<span class="line-number">44</span>
<span class="line-number">45</span>
<span class="line-number">46</span>
<span class="line-number">47</span>
<span class="line-number">48</span>
<span class="line-number">49</span>
<span class="line-number">50</span>
<span class="line-number">51</span>
<span class="line-number">52</span>
<span class="line-number">53</span>
<span class="line-number">54</span>
<span class="line-number">55</span>
<span class="line-number">56</span>
<span class="line-number">57</span>
<span class="line-number">58</span>
<span class="line-number">59</span>
<span class="line-number">60</span>
<span class="line-number">61</span>
<span class="line-number">62</span>
<span class="line-number">63</span>
<span class="line-number">64</span>
<span class="line-number">65</span>
<span class="line-number">66</span>
<span class="line-number">67</span>
<span class="line-number">68</span>
</pre></td><td class="code"><pre><code class="scala"><span class="line"><span class="c1">// ***********************************************</span>
</span><span class="line"><span class="c1">// STEP 1. Load the adjacency graph into a matrix.</span>
</span><span class="line"><span class="c1">// ***********************************************</span>
</span><span class="line">
</span><span class="line"><span class="k">val</span> <span class="n">following</span> <span class="k">=</span> <span class="nc">Tsv</span><span class="o">(</span><span class="nc">GraphFilename</span><span class="o">,</span> <span class="o">(</span><span class="-Symbol">'user1</span><span class="o">,</span> <span class="-Symbol">'user2</span><span class="o">,</span> <span class="-Symbol">'weight</span><span class="o">))</span>
</span><span class="line">
</span><span class="line"><span class="c1">// Binary matrix where cell (u1, u2) means that u1 follows u2.</span>
</span><span class="line"><span class="k">val</span> <span class="n">followingMatrix</span> <span class="k">=</span>
</span><span class="line"> <span class="n">following</span><span class="o">.</span><span class="n">toMatrix</span><span class="o">[</span><span class="kt">Int</span>,<span class="kt">Int</span>,<span class="kt">Double</span><span class="o">](</span><span class="-Symbol">'user1</span><span class="o">,</span> <span class="-Symbol">'user2</span><span class="o">,</span> <span class="-Symbol">'weight</span><span class="o">)</span>
</span><span class="line">
</span><span class="line"><span class="c1">// Binary matrix where cell (u1, u2) means that u1 is followed by u2. </span>
</span><span class="line"><span class="k">val</span> <span class="n">followerMatrix</span> <span class="k">=</span> <span class="n">followingMatrix</span><span class="o">.</span><span class="n">transpose</span>
</span><span class="line">
</span><span class="line"><span class="c1">// Note: we could also form this adjacency matrix differently, by placing</span>
</span><span class="line"><span class="c1">// different weights on the following vs. follower edges.</span>
</span><span class="line"><span class="k">val</span> <span class="n">undirectedAdjacencyMatrix</span> <span class="k">=</span>
</span><span class="line"> <span class="o">(</span><span class="n">followingMatrix</span> <span class="o">+</span> <span class="n">followerMatrix</span><span class="o">).</span><span class="n">rowL1Normalize</span>
</span><span class="line">
</span><span class="line"><span class="c1">// Create a diagonal users matrix (to be used in the "teleportation back</span>
</span><span class="line"><span class="c1">// home" step).</span>
</span><span class="line"><span class="k">val</span> <span class="n">usersMatrix</span> <span class="k">=</span>
</span><span class="line"> <span class="n">following</span><span class="o">.</span><span class="n">unique</span><span class="o">(</span><span class="-Symbol">'user1</span><span class="o">)</span>
</span><span class="line"> <span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="-Symbol">'user1</span> <span class="o">-></span> <span class="o">(</span><span class="-Symbol">'user2</span><span class="o">,</span> <span class="-Symbol">'weight</span><span class="o">))</span> <span class="o">{</span> <span class="n">user1</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=></span> <span class="o">(</span><span class="n">user1</span><span class="o">,</span> <span class="mi">1</span><span class="o">)</span> <span class="o">}</span>
</span><span class="line"> <span class="o">.</span><span class="n">toMatrix</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Int</span>, <span class="kt">Double</span><span class="o">](</span><span class="-Symbol">'user1</span><span class="o">,</span> <span class="-Symbol">'user2</span><span class="o">,</span> <span class="-Symbol">'weight</span><span class="o">)</span>
</span><span class="line">
</span><span class="line"><span class="c1">// ***************************************************</span>
</span><span class="line"><span class="c1">// STEP 2. Compute the personalized PageRank scores.</span>
</span><span class="line"><span class="c1">// See http://nlp.stanford.edu/projects/pagerank.shtml</span>
</span><span class="line"><span class="c1">// for more information on personalized PageRank.</span>
</span><span class="line"><span class="c1">// ***************************************************</span>
</span><span class="line">
</span><span class="line"><span class="c1">// Compute personalized PageRank by running for three iterations,</span>
</span><span class="line"><span class="c1">// and output the top candidates.</span>
</span><span class="line"><span class="k">val</span> <span class="n">pprScores</span> <span class="k">=</span> <span class="n">personalizedPageRank</span><span class="o">(</span><span class="n">usersMatrix</span><span class="o">,</span> <span class="n">undirectedAdjacencyMatrix</span><span class="o">,</span> <span class="n">usersMatrix</span><span class="o">,</span> <span class="mf">0.5</span><span class="o">,</span> <span class="mi">3</span><span class="o">)</span>
</span><span class="line"><span class="n">pprScores</span><span class="o">.</span><span class="n">topRowElems</span><span class="o">(</span><span class="n">numCandidates</span><span class="o">).</span><span class="n">write</span><span class="o">(</span><span class="nc">Tsv</span><span class="o">(</span><span class="nc">OutputFilename</span><span class="o">))</span>
</span><span class="line">
</span><span class="line"><span class="cm">/**</span>
</span><span class="line"><span class="cm"> * Performs a personalized PageRank iteration. The ith row contains the</span>
</span><span class="line"><span class="cm"> * personalized PageRank probabilities around node i.</span>
</span><span class="line"><span class="cm"> *</span>
</span><span class="line"><span class="cm"> * Note the interpretation: </span>
</span><span class="line"><span class="cm"> * - with probability 1 - alpha, we go back to where we started.</span>
</span><span class="line"><span class="cm"> * - with probability alpha, we go to a neighbor.</span>
</span><span class="line"><span class="cm"> *</span>
</span><span class="line"><span class="cm"> * Parameters:</span>
</span><span class="line"><span class="cm"> * </span>
</span><span class="line"><span class="cm"> * startMatrix - a (usually diagonal) matrix, where the ith row specifies</span>
</span><span class="line"><span class="cm"> * where the ith node teleports back to.</span>
</span><span class="line"><span class="cm"> * adjacencyMatrix</span>
</span><span class="line"><span class="cm"> * prevMatrix - a matrix whose ith row contains the personalized PageRank</span>
</span><span class="line"><span class="cm"> * probabilities around the ith node.</span>
</span><span class="line"><span class="cm"> * alpha - the probability of moving to a neighbor (as opposed to</span>
</span><span class="line"><span class="cm"> * teleporting back to the start).</span>
</span><span class="line"><span class="cm"> * numIterations - the number of personalized PageRank iterations to run. </span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"><span class="k">def</span> <span class="n">personalizedPageRank</span><span class="o">(</span><span class="n">startMatrix</span><span class="k">:</span> <span class="kt">Matrix</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Int</span>, <span class="kt">Double</span><span class="o">],</span>
</span><span class="line"> <span class="n">adjacencyMatrix</span><span class="k">:</span> <span class="kt">Matrix</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Int</span>, <span class="kt">Double</span><span class="o">],</span>
</span><span class="line"> <span class="n">prevMatrix</span><span class="k">:</span> <span class="kt">Matrix</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Int</span>, <span class="kt">Double</span><span class="o">],</span>
</span><span class="line"> <span class="n">alpha</span><span class="k">:</span> <span class="kt">Double</span><span class="o">,</span>
</span><span class="line"> <span class="n">numIterations</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">Matrix</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">Int</span>, <span class="kt">Double</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
</span><span class="line"> <span class="k">if</span> <span class="o">(</span><span class="n">numIterations</span> <span class="o"><=</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
</span><span class="line"> <span class="n">prevMatrix</span>
</span><span class="line"> <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
</span><span class="line"> <span class="k">val</span> <span class="n">updatedMatrix</span> <span class="k">=</span> <span class="n">startMatrix</span> <span class="o">*</span> <span class="o">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">alpha</span><span class="o">)</span> <span class="o">+</span>
</span><span class="line"> <span class="o">(</span><span class="n">prevMatrix</span> <span class="o">*</span> <span class="n">adjacencyMatrix</span><span class="o">)</span> <span class="o">*</span> <span class="n">alpha</span>
</span><span class="line"> <span class="n">personalizedPageRank</span><span class="o">(</span><span class="n">startMatrix</span><span class="o">,</span> <span class="n">adjacencyMatrix</span><span class="o">,</span> <span class="n">updatedMatrix</span><span class="o">,</span> <span class="n">alpha</span><span class="o">,</span> <span class="n">numIterations</span> <span class="o">-</span> <span class="mi">1</span><span class="o">)</span>
</span><span class="line"> <span class="o">}</span>
</span><span class="line"><span class="o">}</span>
</span></code></pre></td></tr></table></div>
<div class="clearboth"></div>
<p>Not only is this matrix formulation a more natural way of expressing the algorithm, but since Scalding (by way of Cascading) supports both local and distributed modes, this code runs just as easily on a Hadoop cluster of thousands of machines (assuming our social network is orders of magnitude larger than the one in the contest) as on a sample of data in a laptop. Big data, big matrix style, BOOM.</p>
<h2>Cosine Similarity as L2-Normalized Multiplication</h2>
<p>Here’s another example. Calculating cosine similarity between all users is a natural fit for a matrix formulation since, after all, the cosine similarity between two vectors is just their L2-normalized dot product:</p>
<figcaption><span>Cosine Similarity, Matrix Style</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
</pre></td><td class="code"><pre><code class="scala"><span class="line"><span class="c1">// A matrix where the cell (i, j) is 1 iff user i is followed by user j.</span>
</span><span class="line"><span class="k">val</span> <span class="n">followerMatrix</span> <span class="k">=</span> <span class="o">...</span>
</span><span class="line">
</span><span class="line"><span class="c1">// A matrix where cell (i, j) holds the cosine similarity between</span>
</span><span class="line"><span class="c1">// user i and user j, when both are represented as sets of their followers.</span>
</span><span class="line"><span class="k">val</span> <span class="n">followerBasedSimilarityMatrix</span> <span class="k">=</span>
</span><span class="line"> <span class="n">followerMatrix</span><span class="o">.</span><span class="n">rowL2Normalize</span> <span class="o">*</span> <span class="n">followerMatrix</span><span class="o">.</span><span class="n">rowL2Normalize</span><span class="o">.</span><span class="n">transpose</span>
</span></code></pre></td></tr></table></div>
<h2>A Similarity Extension</h2>
<p>But let’s go one step further.</p>
<p>To change examples for ease of exposition: suppose you’ve bought a bunch of books on Amazon, and Amazon wants to recommend a new book you’ll like. Since Amazon knows similarities between all pairs of books, one natural way to generate this recommendation is to:</p>
<ol>
<li>Take every book B.</li>
<li>Calculate the similarity between B and each book you bought.</li>
<li>Sum up all these similarities to get your recommendation score for B.</li>
</ol>
<p>In other words, the recommendation score for book B on user U is:</p>
<p>DidUserBuy(U, Book 1) * SimilarityBetween(Book B, Book 1) + DidUserBuy(U, Book 2) * SimilarityBetween(Book B, Book2) + … + DidUserBuy(U, Book n) * SimilarityBetween(Book B, Book n)</p>
<p>This, too, is a dot product! So it can also be rewritten as a matrix multiplication:</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
</pre></td><td class="code"><pre><code class="scala"><span class="line"><span class="c1">// A matrix where cell (i, j) holds the similarity between books i and j.</span>
</span><span class="line"><span class="k">val</span> <span class="n">bookSimilarityMatrix</span> <span class="k">=</span> <span class="o">...</span>
</span><span class="line">
</span><span class="line"><span class="c1">// A matrix where cell (i, j) is 1 if user i has bought book j, </span>
</span><span class="line"><span class="c1">// and 0 otherwise.</span>
</span><span class="line"><span class="k">val</span> <span class="n">userPurchaseMatrix</span> <span class="k">=</span> <span class="o">...</span>
</span><span class="line">
</span><span class="line"><span class="c1">// A matrix where cell (i, j) holds the recommendation score of</span>
</span><span class="line"><span class="c1">// book j to user i.</span>
</span><span class="line"><span class="k">val</span> <span class="n">recommendationMatrix</span> <span class="k">=</span> <span class="n">userPurchaseMatrix</span> <span class="o">*</span> <span class="n">bookSimilarityMatrix</span>
</span></code></pre></td></tr></table></div>
<p>Of course, there’s a natural analogy between this score and the feature I described a while back above, where I compute a similarity score between a destination node and a source node’s followees (when all nodes are represented as sets of followers):</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
<span class="line-number">19</span>
<span class="line-number">20</span>
<span class="line-number">21</span>
<span class="line-number">22</span>
<span class="line-number">23</span>
<span class="line-number">24</span>
<span class="line-number">25</span>
<span class="line-number">26</span>
</pre></td><td class="code"><pre><code class="scala"><span class="line"><span class="cm">/**</span>
</span><span class="line"><span class="cm"> * Iterate over each of user1's followings, compute their similarity</span>
</span><span class="line"><span class="cm"> * with user2 when both are represented as sets of followers, and return</span>
</span><span class="line"><span class="cm"> * the sum of these similarities.</span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"><span class="k">def</span> <span class="n">followerBasedSimilarityToFollowings</span><span class="o">(</span><span class="n">user1</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">user2</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span>
</span><span class="line"> <span class="o">(</span><span class="k">implicit</span> <span class="n">similarity</span><span class="k">:</span> <span class="kt">SimilarityMetric</span><span class="o">[</span><span class="kt">Int</span><span class="o">])</span><span class="k">:</span> <span class="kt">Double</span> <span class="o">=</span> <span class="o">{</span>
</span><span class="line"> <span class="n">getFollowingsWithout</span><span class="o">(</span><span class="n">user1</span><span class="o">,</span> <span class="n">user2</span><span class="o">)</span>
</span><span class="line"> <span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">similarityByFollowers</span><span class="o">(</span><span class="k">_</span><span class="o">,</span> <span class="n">user2</span><span class="o">)(</span><span class="n">similarity</span><span class="o">)</span> <span class="o">}</span>
</span><span class="line"> <span class="o">.</span><span class="n">sum</span>
</span><span class="line"><span class="o">}</span>
</span><span class="line">
</span><span class="line"><span class="cm">/**</span>
</span><span class="line"><span class="cm"> * The matrix version of the above function.</span>
</span><span class="line"><span class="cm"> *</span>
</span><span class="line"><span class="cm"> * Why are these the same? Note that the above function simply computes:</span>
</span><span class="line"><span class="cm"> * DoesUserFollow(User A, User 1) * Similarity(User 1, User B) + </span>
</span><span class="line"><span class="cm"> * DoesUserFollow(User A, User 2) * Similarity(User 2, User B) + ... + </span>
</span><span class="line"><span class="cm"> * DoesUserFollow(User A, User n) * Similarity(User n, User B)</span>
</span><span class="line"><span class="cm"> */</span>
</span><span class="line"><span class="k">val</span> <span class="n">followingMatrix</span> <span class="k">=</span> <span class="o">...</span>
</span><span class="line"><span class="k">val</span> <span class="n">followerBasedSimilarityMatrix</span> <span class="k">=</span>
</span><span class="line"> <span class="n">followerMatrix</span><span class="o">.</span><span class="n">rowL2Normalize</span> <span class="o">*</span> <span class="n">followerMatrix</span><span class="o">.</span><span class="n">rowL2Normalize</span><span class="o">.</span><span class="n">transpose</span>
</span><span class="line">
</span><span class="line"><span class="k">val</span> <span class="n">followerBasedSimilarityToFollowingsMatrix</span> <span class="k">=</span>
</span><span class="line"> <span class="n">followingMatrix</span> <span class="o">*</span> <span class="n">followerBasedSimilarityMatrix</span>
</span></code></pre></td></tr></table></div>
<p>For people comfortable expressing their computations in a vector manner, writing your computations as matrix manipulations often makes experimenting with different algorithms much more fluid. Imagine, for example, that you want to switch from L1 normalization to L2 normalization, or that you want to express your objects as binary sets rather than weighted vectors. Both of these become simple one-line changes when you have vectors and matrices as first-class objects, but are much more tedious (<em>especially in a MapReduce land where this matrix library was designed to be applied!</em>) when you don’t.</p>
<h1>Finish Line</h1>
<p>By now, I think I’ve spent more time writing this post than on the contest itself, so let’s wrap up.</p>
<p>I often get asked what kinds of tools I like to use, so for this competition my kit consisted of:</p>
<ul>
<li>Scala, for code that needed to be fast (e.g., extracting features) or that I was going to run repeatedly (e.g., scoring my validation set).</li>
<li>Python, for my machine learning models, because <a href="http://scikit-learn.org/stable/">scikit-learn</a> is awesome.</li>
<li>Ruby, for quick one-off scripts.</li>
<li>R, for some data analysis and simple plotting.</li>
<li>Coffeescript and d3, for the interactive visualizations.</li>
</ul>
<p>Finally, I put up a <a href="https://github.com/echen/link-prediction">Github repository</a> containing some code, and here are a couple other posts I’ve written that people who like this entry might also enjoy:</p>
<ul>
<li><a href="http://blog.echen.me/2011/09/07/information-transmission-in-a-social-network-dissecting-the-spread-of-a-quora-post/">Information transmission in a social network</a>, a case study in how information propagates through a social graph.</li>
<li><a href="http://blog.echen.me/2012/02/09/movie-recommendations-and-more-via-mapreduce-and-scalding/">Movie recommendations in Scalding</a>, Twitter’s Scala-based Hadoop framework built on top of Cascading.</li>
<li><a href="http://blog.echen.me/2011/10/24/winning-the-netflix-prize-a-summary/">A summary of the algorithms behind the Netflix Prize</a>, another crowdsourced recommendation contest for predicting movie ratings.</li>
</ul>
</div>
Soda vs. Pop with Twitter2012-07-06T04:15:00+02:002012-07-06T04:15:00+02:00Edwin Chentag:blog.echen.me,2012-07-06:/2012/07/06/soda-vs-pop-with-twitter/
<div class="entry-content"><p>One of the great things about Twitter is that it’s a global conversation anyone can join anytime. Eavesdropping on the world, what what!</p>
<p>Of course, it gets even better when you can <em>mine</em> all this chatter to study the way humans live and interact.</p>
<p>For example, <a href="http://blog.echen.me/2011/04/18/twifferences-between-californians-and-new-yorkers/">how do people …</a></p></div>
<div class="entry-content"><p>One of the great things about Twitter is that it’s a global conversation anyone can join anytime. Eavesdropping on the world, what what!</p>
<p>Of course, it gets even better when you can <em>mine</em> all this chatter to study the way humans live and interact.</p>
<p>For example, <a href="http://blog.echen.me/2011/04/18/twifferences-between-californians-and-new-yorkers/">how do people in New York City differ from those in Silicon Valley?</a> We tend to think they’re more financially driven and restless with the world – is this true, and if so, <a href="http://blog.echen.me/2011/04/18/twifferences-between-californians-and-new-yorkers/">how much more</a>?</p>
<p>Or how does language change as you travel to different regions? Recall the classic soda vs. pop. vs. coke question: some people use the word “soda” to describe their soft drinks, others use “pop”, and still others use “coke”. Who says what where?</p>
<p>Let’s take a look.</p>
<p><a href="http://i.imgur.com/OgNjpqI.png"><img src="http://i.imgur.com/OgNjpqI.png" alt="United States" /></a></p>
<p>To make this map, I sampled geo-tagged tweets containing the words “soda”, “pop”, or “coke”, performed some state-of-the-art NLP technology to ensure the tweets were soft drink related (e.g., the tweets had to contain “drink soda” or “drink a pop”), and tried to filter out coke tweets that were specifically about the Coke brand (e.g., Coke Zero).</p>
<p>It’s a little cluttered, though, so let’s clean it up by aggregating nearby tweets.</p>
<p><a href="http://i.imgur.com/iWesmoA.png"><img src="http://i.imgur.com/iWesmoA.png" alt="United States Binned" /></a></p>
<p>Here, I bucketed all tweets within a 0.333 latitude/longitude radius, calculated the term distribution within each bucket, and colored each bucket with the word furthest from its overall mean. I also sized each point according to the (log-transformed) number of tweets in the bucket.</p>
<p>We can see that:</p>
<ul>
<li>The South is pretty Coke-heavy.</li>
<li>Soda belongs to the Northeast and far West.</li>
<li>Pop gets the mid-West, except for some interesting spots of blue around Wisconsin and the Illinois-Missouri border.</li>
</ul>
<p>For comparison, here’s another map based on a survey at <a href="http://www.popvssoda.com/">popvssoda.com</a>.</p>
<p><a href="http://i.imgur.com/5jAbC0G.png"><img src="http://i.imgur.com/5jAbC0G.png" alt="Pop vs. Soda Map" /></a></p>
<p>We can see similar patterns, though interestingly, our map has less Coke in the Southeast and less pop in the Northwest.</p>
<p>Finally, here’s a world map of the terms, bucketed again. Notice that “pop” seems to be prevalent only in parts of the United States and Canada.</p>
<p><a href="http://i.imgur.com/GewG65x.png"><img src="http://i.imgur.com/GewG65x.png" alt="World" /></a></p>
<p>As some astute readers noted, though, the seeming dominance of coke is probably due to the difficulty in distinguishing the generic use of coke for soft drinks in general from the particular use of coke for referring to the Coca-Cola brand.</p>
<p>So let’s instead look at a world map of a couple other soft drink terms (“fizzy drink”, “mineral”, and “tonic”):</p>
<p><a href="http://i.imgur.com/R0CASuw.png"><img src="http://i.imgur.com/R0CASuw.png" alt="Fizzy Drink vs. Mineral vs. Tonic" /></a></p>
<p>Notice that:</p>
<ul>
<li>“Fizzy drink” shows up for the UK, New Zealand, and Maine.</li>
<li>“Tonic” appears in Massachusetts.</li>
<li>While South Africa gets “fizzy drink”, Nigeria gets “mineral”.</li>
</ul>
<p>I’ve been getting a lot of questions lately about interesting things you can do with the Twitter API, so this was just one small project I’ve worked on to illustrate. <a href="http://www.cc.gatech.edu/~jeisenst/papers/emnlp2010.pdf">This paper</a> contains another awesome application of Twitter data to geographic language variation, and just for fun, here are a few other cute mini-projects:</p>
<p>What do people eat during the Super Bowl? (wings and beer, apparently)</p>
<p><a href="https://twitter.com/echen/status/166343879547822080"><img src="http://i.imgur.com/6D3D4OJ.png" alt="Superbowl Snacks" /></a></p>
<p>What do people want for Christmas, compared to what they actually get?</p>
<p><a href="https://twitter.com/echen/status/153683967315419136"><img src="http://i.imgur.com/rJmfMTA.png" alt="Christmas" /></a></p>
<p>What do guys and girls <em>really</em> say?</p>
<p><a href="https://twitter.com/echen/status/261667822793551873/photo/1"><img src="http://i.imgur.com/awPZdyj.png" alt="Shit Guys and Girls Say" /></a></p>
<p>When were people losing and gaining power during Hurricane Sandy? (<a href="http://blog.echen.me/hurricane-sandy-outages/">click</a> the image to interact)</p>
<p><a href="http://blog.echen.me/hurricane-sandy-outages/"><img src="http://i.imgur.com/6MzIHww.png" alt="Sandy Power Outages" /></a></p>
<p>How does information of a <em>geographic</em>-specific nature spread? (<a href="http://hurricanesandy.herokuapp.com/">click</a> the image to see a dynamic visualization of when and where tweets related to surviving Hurricane Sandy were shared)</p>
<p><a href="http://hurricanesandy.herokuapp.com/"><img src="http://i.imgur.com/x3Z2Kpe.png" alt="Hurricane Sandy Retweets" /></a></p>
<p>Can we use Twitter to measure presidential votes? (yes!)</p>
<p><a href="https://twitter.com/echen/status/265894918382305284/photo/1"><img src="http://i.imgur.com/D7GhNLR.png" alt="Electoral Map" /></a></p>
</div>
Infinite Mixture Models with Nonparametric Bayes and the Dirichlet Process2012-03-20T04:15:00+01:002012-03-20T04:15:00+01:00Edwin Chentag:blog.echen.me,2012-03-20:/2012/03/20/infinite-mixture-models-with-nonparametric-bayes-and-the-dirichlet-process/
<div class="entry-content"><p>Imagine you’re a budding chef. A data-curious one, of course, so you start by taking a set of foods (pizza, salad, spaghetti, etc.) and ask 10 friends how much of each they ate in the past day.</p>
<p>Your goal: to find natural <em>groups</em> of foodies, so that you can …</p></div>
<div class="entry-content"><p>Imagine you’re a budding chef. A data-curious one, of course, so you start by taking a set of foods (pizza, salad, spaghetti, etc.) and ask 10 friends how much of each they ate in the past day.</p>
<p>Your goal: to find natural <em>groups</em> of foodies, so that you can better cater to each cluster’s tastes. For example, your fratboy friends might love <a href="https://twitter.com/#!/edchedch/status/166343879547822080">wings and beer</a>, your anime friends might love soba and sushi, your hipster friends probably dig tofu, and so on.</p>
<p>So how can you use the data you’ve gathered to discover different kinds of groups?</p>
<p><a href="http://i.imgur.com/sxBfR1R.png"><img src="http://i.imgur.com/sxBfR1R.png" alt="Clustering Example" /></a></p>
<p>One way is to use a standard clustering algorithm like <strong>k-means</strong> or <strong>Gaussian mixture modeling</strong> (see <a href="http://blog.echen.me/2011/03/19/counting-clusters/">this previous post</a> for a brief introduction). The problem is that these both assume a <em>fixed</em> number of clusters, which they need to be told to find. There are a couple methods for selecting the number of clusters to learn (e.g., the <a href="http://blog.echen.me/2011/03/19/counting-clusters/">gap and prediction strength statistics</a>), but the problem is a more fundamental one: most real-world data simply doesn’t have a fixed number of clusters.</p>
<p>That is, suppose we’ve asked 10 of our friends what they ate in the past day, and we want to find groups of eating preferences. There’s really an infinite number of foodie types (carnivore, vegan, snacker, Italian, healthy, fast food, heavy eaters, light eaters, and so on), but with only 10 friends, we simply don’t have enough data to detect them all. (Indeed, we’re limited to 10 clusters!) So whereas k-means starts with the incorrect assumption that there’s a fixed, finite number of clusters that our points come from, <em>no matter if we feed it more data</em>, what we’d really like is a method positing an infinite number of hidden clusters that naturally arise as we ask more friends about their food habits. (For example, with only 2 data points, we might not be able to tell the difference between vegans and vegetarians, but with 200 data points, we probably could.)</p>
<p>Luckily for us, this is precisely the purview of <strong>nonparametric Bayes</strong>.*</p>
<p>*Nonparametric Bayes refers to a class of techniques that allow some parameters to change with the data. In our case, for example, instead of fixing the number of clusters to be discovered, we allow it to grow as more data comes in.</p>
<h1>A Generative Story</h1>
<p>Let’s describe a generative model for finding clusters in any set of data. We assume an infinite set of latent groups, where each group is described by some set of parameters. For example, each group could be a Gaussian with a specified mean $\mu_i$ and standard deviation $\sigma_i$, and these group parameters themselves are assumed to come from some base distribution $G_0$. Data is then generated in the following manner:</p>
<ul>
<li>Select a cluster.</li>
<li>Sample from that cluster to generate a new point.</li>
</ul>
<p>(Note the resemblance to a <a href="http://en.wikipedia.org/wiki/Mixture_model">finite mixture model</a>.)</p>
<p>For example, suppose we ask 10 friends how many calories of pizza, salad, and rice they ate yesterday. Our groups could be:</p>
<ul>
<li>A Gaussian centered at (pizza = 5000, salad = 100, rice = 500) (i.e., a pizza lovers group).</li>
<li>A Gaussian centered at (pizza = 100, salad = 3000, rice = 1000) (maybe a vegan group).</li>
<li>A Gaussian centered at (pizza = 100, salad = 100, rice = 10000) (definitely Asian).</li>
<li>…</li>
</ul>
<p>When deciding what to eat when she woke up yesterday, Alice could have thought <em>girl, I’m in the mood for pizza</em> and her food consumption yesterday would have been a sample from the pizza Gaussian. Similarly, Bob could have spent the day in Chinatown, thereby sampling from the Asian Gaussian for his day’s meals. And so on.</p>
<p>The big question, then, is: how do we assign each friend to a group?</p>
<h1>Assigning Groups</h1>
<h2>Chinese Restaurant Process</h2>
<p>One way to assign friends to groups is to use a <strong>Chinese Restaurant Process</strong>. This works as follows: Imagine a restaurant where all your friends went to eat yesterday…</p>
<ul>
<li>Initially the restaurant is empty.</li>
<li>The first person to enter (Alice) sits down at a table (selects a group). She then orders food for the table (i.e., she selects parameters for the group); everyone else who joins the table will then be limited to eating from the food she ordered.</li>
<li>The second person to enter (Bob) sits down at a table. Which table does he sit at? With probability $\alpha / (1 + \alpha)$ he sits down at a new table (i.e., selects a new group) and orders food for the table; with probability $1 / (1 + \alpha)$ he sits with Alice and eats from the food she’s already ordered (i.e., he’s in the same group as Alice).</li>
<li>…</li>
<li>The (n+1)-st person sits down at a new table with probability $\alpha / (n + \alpha)$, and at table k with probability $n_k / (n + \alpha)$, where $n_k$ is the number of people currently sitting at table k.</li>
</ul>
<p>Note a couple things:</p>
<ul>
<li>The more people (data points) there are at a table (cluster), the more likely it is that people (new data points) will join it. In other words, our groups satisfy a <strong>rich get richer</strong> property.</li>
<li>There’s always a small probability that someone joins an entirely new table (i.e., a new group is formed).</li>
<li>The probability of a new group depends on $\alpha$. So we can think of $\alpha$ as a <strong>dispersion parameter</strong> that affects the dispersion of our datapoints. The lower alpha is, the more tightly clustered our data points; the higher it is, the more clusters we have in any finite set of points.</li>
</ul>
<p>(Also notice the resemblance between table selection probabilities and a Dirichlet distribution…)</p>
<p>Just to summarize, given n data points, the Chinese Restaurant Process specifies a distribution over partitions (table assignments) of these points. We can also generate parameters for each partition/table from a base distribution $G_0$ (for example, each table could represent a Gaussian whose mean and standard deviation are sampled from $G_0$), though to be clear, this is not part of the CRP itself.</p>
<h3>Code</h3>
<p>Since code makes everything better, here’s some Ruby to simulate a CRP:</p>
<figcaption><span>Chinese Restaurant Process </span><a href="https://github.com/echen/dirichlet-process/blob/master/chinese_restaurant_process.rb">chinese_restaurant_process.rb</a></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
<span class="line-number">19</span>
<span class="line-number">20</span>
<span class="line-number">21</span>
<span class="line-number">22</span>
<span class="line-number">23</span>
<span class="line-number">24</span>
<span class="line-number">25</span>
<span class="line-number">26</span>
<span class="line-number">27</span>
</pre></td><td class="code"><pre><code class="ruby"><span class="line"><span class="c1"># Generate table assignments for `num_customers` customers, according to</span>
</span><span class="line"><span class="c1"># a Chinese Restaurant Process with dispersion parameter `alpha`.</span>
</span><span class="line"><span class="c1">#</span>
</span><span class="line"><span class="c1"># returns an array of integer table assignments</span>
</span><span class="line"><span class="k">def</span> <span class="nf">chinese_restaurant_process</span><span class="p">(</span><span class="n">num_customers</span><span class="p">,</span> <span class="n">alpha</span><span class="p">)</span>
</span><span class="line"> <span class="k">return</span> <span class="o">[]</span> <span class="k">if</span> <span class="n">num_customers</span> <span class="o"><=</span> <span class="mi">0</span>
</span><span class="line">
</span><span class="line"> <span class="n">table_assignments</span> <span class="o">=</span> <span class="o">[</span><span class="mi">1</span><span class="o">]</span> <span class="c1"># first customer sits at table 1</span>
</span><span class="line"> <span class="n">next_open_table</span> <span class="o">=</span> <span class="mi">2</span> <span class="c1"># index of the next empty table</span>
</span><span class="line">
</span><span class="line"> <span class="c1"># Now generate table assignments for the rest of the customers.</span>
</span><span class="line"> <span class="mi">1</span><span class="o">.</span><span class="n">upto</span><span class="p">(</span><span class="n">num_customers</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
</span><span class="line"> <span class="k">if</span> <span class="nb">rand</span> <span class="o"><</span> <span class="n">alpha</span><span class="o">.</span><span class="n">to_f</span> <span class="o">/</span> <span class="p">(</span><span class="n">alpha</span> <span class="o">+</span> <span class="n">i</span><span class="p">)</span>
</span><span class="line"> <span class="c1"># Customer sits at new table.</span>
</span><span class="line"> <span class="n">table_assignments</span> <span class="o"><<</span> <span class="n">next_open_table</span>
</span><span class="line"> <span class="n">next_open_table</span> <span class="o">+=</span> <span class="mi">1</span>
</span><span class="line"> <span class="k">else</span>
</span><span class="line"> <span class="c1"># Customer sits at an existing table.</span>
</span><span class="line"> <span class="c1"># He chooses which table to sit at by giving equal weight to each</span>
</span><span class="line"> <span class="c1"># customer already sitting at a table. </span>
</span><span class="line"> <span class="n">which_table</span> <span class="o">=</span> <span class="n">table_assignments</span><span class="o">[</span><span class="nb">rand</span><span class="p">(</span><span class="n">table_assignments</span><span class="o">.</span><span class="n">size</span><span class="p">)</span><span class="o">]</span>
</span><span class="line"> <span class="n">table_assignments</span> <span class="o"><<</span> <span class="n">which_table</span>
</span><span class="line"> <span class="k">end</span>
</span><span class="line"> <span class="k">end</span>
</span><span class="line">
</span><span class="line"> <span class="n">table_assignments</span>
</span><span class="line"><span class="k">end</span>
</span></code></pre></td></tr></table></div>
<p>And here’s some sample output:</p>
<figcaption><span>Chinese Restaurant Process </span><a href="https://github.com/echen/dirichlet-process/blob/master/chinese_restaurant_process.rb">chinese_restaurant_process.rb</a></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
</pre></td><td class="code"><pre><code class="ruby"><span class="line"><span class="o">></span> <span class="n">chinese_restaurant_process</span><span class="p">(</span><span class="n">num_customers</span> <span class="o">=</span> <span class="mi">10</span><span class="p">,</span> <span class="n">alpha</span> <span class="o">=</span> <span class="mi">1</span><span class="p">)</span>
</span><span class="line"><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span> <span class="c1"># table assignments from run 1</span>
</span><span class="line"><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">3</span> <span class="c1"># table assignments from run 2</span>
</span><span class="line"><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span> <span class="c1"># table assignments from run 3</span>
</span><span class="line">
</span><span class="line"><span class="o">></span> <span class="n">chinese_restaurant_process</span><span class="p">(</span><span class="n">num_customers</span> <span class="o">=</span> <span class="mi">10</span><span class="p">,</span> <span class="n">alpha</span> <span class="o">=</span> <span class="mi">3</span><span class="p">)</span>
</span><span class="line"><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span>
</span><span class="line"><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span>
</span><span class="line"><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span>
</span><span class="line">
</span><span class="line"><span class="o">></span> <span class="n">chinese_restaurant_process</span><span class="p">(</span><span class="n">num_customers</span> <span class="o">=</span> <span class="mi">10</span><span class="p">,</span> <span class="n">alpha</span> <span class="o">=</span> <span class="mi">5</span><span class="p">)</span>
</span><span class="line"><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">8</span>
</span><span class="line"><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">7</span>
</span><span class="line"><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span>
</span></code></pre></td></tr></table></div>
<p>Notice that as we increase $\alpha$, so too does the number of distinct tables increase.</p>
<h2>Polya Urn Model</h2>
<p>Another method for assigning friends to groups is to follow the <strong>Polya Urn Model</strong>. This is basically the same model as the Chinese Restaurant Process, just with a different metaphor.</p>
<ul>
<li>We start with an urn containing $\alpha G_0(x)$ balls of “color” $x$, for each possible value of $x$. ($G_0$ is our base distribution, and $G_0(x)$ is the probability of sampling $x$ from $G_0$). Note that these are possibly fractional balls.</li>
<li>At each time step, draw a ball from the urn, note its color, and then drop both the original ball plus a new ball of the same color back into the urn.</li>
</ul>
<p>Note the connection between this process and the CRP: balls correspond to people (i.e., data points), colors correspond to table assignments (i.e., clusters), alpha is again a dispersion parameter (put differently, a prior), colors satisfy a rich-get-richer property (since colors with many balls are more likely to get drawn), and so on. (Again, there’s also a connection between this urn model and <a href="http://en.wikipedia.org/wiki/Dirichlet_distribution#P.C3.B3lya.27s_urn">the urn model for the (finite) Dirichlet distribution</a>…)</p>
<p>To be precise, the difference between the CRP and the Polya Urn Model is that the CRP specifies only a distribution over <em>partitions</em> (i.e., table assignments), but doesn’t assign parameters to each group, whereas the Polya Urn Model does both.</p>
<h3>Code</h3>
<p>Again, here’s some code for simulating a Polya Urn Model:</p>
<figcaption><span>Polya Urn Model </span><a href="https://github.com/echen/dirichlet-process/blob/master/polya_urn_model.rb">polya_urn_model.rb</a></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
<span class="line-number">19</span>
<span class="line-number">20</span>
<span class="line-number">21</span>
<span class="line-number">22</span>
<span class="line-number">23</span>
</pre></td><td class="code"><pre><code class="ruby"><span class="line"><span class="c1"># Draw `num_balls` colored balls according to a Polya Urn Model</span>
</span><span class="line"><span class="c1"># with a specified base color distribution and dispersion parameter</span>
</span><span class="line"><span class="c1"># `alpha`.</span>
</span><span class="line"><span class="c1">#</span>
</span><span class="line"><span class="c1"># returns an array of ball colors</span>
</span><span class="line"><span class="k">def</span> <span class="nf">polya_urn_model</span><span class="p">(</span><span class="n">base_color_distribution</span><span class="p">,</span> <span class="n">num_balls</span><span class="p">,</span> <span class="n">alpha</span><span class="p">)</span>
</span><span class="line"> <span class="k">return</span> <span class="o">[]</span> <span class="k">if</span> <span class="n">num_balls</span> <span class="o"><=</span> <span class="mi">0</span>
</span><span class="line">
</span><span class="line"> <span class="n">balls_in_urn</span> <span class="o">=</span> <span class="o">[]</span>
</span><span class="line"> <span class="mi">0</span><span class="o">.</span><span class="n">upto</span><span class="p">(</span><span class="n">num_balls</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
</span><span class="line"> <span class="k">if</span> <span class="nb">rand</span> <span class="o"><</span> <span class="n">alpha</span><span class="o">.</span><span class="n">to_f</span> <span class="o">/</span> <span class="p">(</span><span class="n">alpha</span> <span class="o">+</span> <span class="n">balls_in_urn</span><span class="o">.</span><span class="n">size</span><span class="p">)</span>
</span><span class="line"> <span class="c1"># Draw a new color, put a ball of this color in the urn.</span>
</span><span class="line"> <span class="n">new_color</span> <span class="o">=</span> <span class="n">base_color_distribution</span><span class="o">.</span><span class="n">call</span>
</span><span class="line"> <span class="n">balls_in_urn</span> <span class="o"><<</span> <span class="n">new_color</span>
</span><span class="line"> <span class="k">else</span>
</span><span class="line"> <span class="c1"># Draw a ball from the urn, add another ball of the same color.</span>
</span><span class="line"> <span class="n">ball</span> <span class="o">=</span> <span class="n">balls_in_urn</span><span class="o">[</span><span class="nb">rand</span><span class="p">(</span><span class="n">balls_in_urn</span><span class="o">.</span><span class="n">size</span><span class="p">)</span><span class="o">]</span>
</span><span class="line"> <span class="n">balls_in_urn</span> <span class="o"><<</span> <span class="n">ball</span>
</span><span class="line"> <span class="k">end</span>
</span><span class="line"> <span class="k">end</span>
</span><span class="line">
</span><span class="line"> <span class="n">balls_in_urn</span>
</span><span class="line"><span class="k">end</span>
</span></code></pre></td></tr></table></div>
<p>And here’s some sample output, using a uniform distribution over the unit interval as the color distribution to sample from:</p>
<figcaption><span>Polya Urn Model </span><a href="https://github.com/echen/dirichlet-process/blob/master/polya_urn_model.rb">polya_urn_model.rb</a></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
</pre></td><td class="code"><pre><code class="ruby"><span class="line"><span class="o">></span> <span class="n">unit_uniform</span> <span class="o">=</span> <span class="nb">lambda</span> <span class="p">{</span> <span class="p">(</span><span class="nb">rand</span> <span class="o">*</span> <span class="mi">100</span><span class="p">)</span><span class="o">.</span><span class="n">to_i</span> <span class="o">/</span> <span class="mi">100</span><span class="o">.</span><span class="mi">0</span> <span class="p">}</span>
</span><span class="line">
</span><span class="line"><span class="o">></span> <span class="n">polya_urn_model</span><span class="p">(</span><span class="n">unit_uniform</span><span class="p">,</span> <span class="n">num_balls</span> <span class="o">=</span> <span class="mi">10</span><span class="p">,</span> <span class="n">alpha</span> <span class="o">=</span> <span class="mi">1</span><span class="p">)</span>
</span><span class="line"><span class="mi">0</span><span class="o">.</span><span class="mi">27</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">89</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">89</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">89</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">73</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">98</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">43</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">98</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">89</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">53</span> <span class="c1"># colors in the urn from run 1</span>
</span><span class="line"><span class="mi">0</span><span class="o">.</span><span class="mi">26</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">26</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">46</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">26</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">26</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">26</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">26</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">26</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">26</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">85</span> <span class="c1"># colors in the urn from run 2</span>
</span><span class="line"><span class="mi">0</span><span class="o">.</span><span class="mi">96</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">87</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">96</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">87</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">96</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">96</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">87</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">96</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">96</span><span class="p">,</span> <span class="mi">0</span><span class="o">.</span><span class="mi">96</span> <span class="c1"># colors in the urn from run 3</span>
</span></code></pre></td></tr></table></div>
<h3>Code, Take 2</h3>
<p>Here’s the same code for a Polya Urn Model, but in R:</p>
<figcaption><span>Polya Urn Model </span><a href="https://github.com/echen/dirichlet-process/blob/master/polya_urn_model.R">polya_urn_model.R</a></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
<span class="line-number">19</span>
<span class="line-number">20</span>
</pre></td><td class="code"><pre><code class="r"><span class="line"><span class="c1"># Return a vector of `num_balls` ball colors according to a Polya Urn Model</span>
</span><span class="line"><span class="c1"># with dispersion `alpha`, sampling from a specified base color distribution.</span>
</span><span class="line">polya_urn_model <span class="o">=</span> <span class="kr">function</span><span class="p">(</span>base_color_distribution<span class="p">,</span> num_balls<span class="p">,</span> alpha<span class="p">)</span> <span class="p">{</span>
</span><span class="line"> balls <span class="o">=</span> c<span class="p">()</span>
</span><span class="line">
</span><span class="line"> <span class="kr">for</span> <span class="p">(</span>i in <span class="m">1</span>:num_balls<span class="p">)</span> <span class="p">{</span>
</span><span class="line"> <span class="kr">if</span> <span class="p">(</span>runif<span class="p">(</span><span class="m">1</span><span class="p">)</span> <span class="o"><</span> alpha <span class="o">/</span> <span class="p">(</span>alpha <span class="o">+</span> length<span class="p">(</span>balls<span class="p">)))</span> <span class="p">{</span>
</span><span class="line"> <span class="c1"># Add a new ball color.</span>
</span><span class="line"> new_color <span class="o">=</span> base_color_distribution<span class="p">()</span>
</span><span class="line"> balls <span class="o">=</span> c<span class="p">(</span>balls<span class="p">,</span> new_color<span class="p">)</span>
</span><span class="line"> <span class="p">}</span> <span class="kr">else</span> <span class="p">{</span>
</span><span class="line"> <span class="c1"># Pick out a ball from the urn, and add back a</span>
</span><span class="line"> <span class="c1"># ball of the same color.</span>
</span><span class="line"> ball <span class="o">=</span> balls<span class="p">[</span>sample<span class="p">(</span><span class="m">1</span>:length<span class="p">(</span>balls<span class="p">),</span> <span class="m">1</span><span class="p">)]</span>
</span><span class="line"> balls <span class="o">=</span> c<span class="p">(</span>balls<span class="p">,</span> ball<span class="p">)</span>
</span><span class="line"> <span class="p">}</span>
</span><span class="line"> <span class="p">}</span>
</span><span class="line">
</span><span class="line"> balls
</span><span class="line"><span class="p">}</span>
</span></code></pre></td></tr></table></div>
<p>Here are some sample density plots of the colors in the urn, when using a unit normal as the base color distribution:</p>
<p><a href="http://i.imgur.com/4aOd7TI.png"><img src="http://i.imgur.com/4aOd7TI.png" alt="Polya Urn Model, Alpha = 1" /></a></p>
<p><a href="http://i.imgur.com/vTLeppf.png"><img src="http://i.imgur.com/vTLeppf.png" alt="Polya Urn Model, Alpha = 5" /></a></p>
<p><a href="http://i.imgur.com/SQPZY95.png"><img src="http://i.imgur.com/SQPZY95.png" alt="Polya Urn Model, Alpha = 25" /></a></p>
<p><a href="http://i.imgur.com/PFmE0q3.png"><img src="http://i.imgur.com/PFmE0q3.png" alt="Polya Urn Model, Alpha = 50" /></a></p>
<p>Notice that as alpha increases (i.e., we sample more new ball colors from our base; i.e., as we place more weight on our prior), the colors in the urn tend to a unit normal (our base color distribution).</p>
<p>And here are some sample plots of points generated by the urn, for varying values of alpha:</p>
<ul>
<li>Each color in the urn is sampled from a uniform distribution over [0,10]x[0,10] (i.e., a [0, 10] square).</li>
<li>Each group is a Gaussian with standard deviation 0.1 and mean equal to its associated color, and these Gaussian groups generate points.</li>
</ul>
<p><a href="http://i.imgur.com/QdRnCbn.png"><img src="http://i.imgur.com/QdRnCbn.png" alt="Alpha 0.1" /></a></p>
<p><a href="http://i.imgur.com/n0CLgfs.png"><img src="http://i.imgur.com/n0CLgfs.png" alt="Alpha 0.2" /></a></p>
<p><a href="http://i.imgur.com/yp9XINd.png"><img src="http://i.imgur.com/yp9XINd.png" alt="Alpha 0.3" /></a></p>
<p><a href="http://i.imgur.com/MkoA8kP.png"><img src="http://i.imgur.com/MkoA8kP.png" alt="Alpha 0.5" /></a></p>
<p><a href="http://i.imgur.com/UEmRqQU.png"><img src="http://i.imgur.com/UEmRqQU.png" alt="Alpha 1.0" /></a></p>
<p>Notice that the points clump together in fewer clusters for low values of alpha, but become more dispersed as alpha increases.</p>
<h2>Stick-Breaking Process</h2>
<p>Imagine running either the Chinese Restaurant Process or the Polya Urn Model without stop. For each group $i$, this gives a proportion $w_i$ of points that fall into group $i$.</p>
<p>So instead of running the CRP or Polya Urn model to figure out these proportions, can we simply generate them directly?</p>
<p>This is exactly what the Stick-Breaking Process does:</p>
<ul>
<li>Start with a stick of length one.</li>
<li>Generate a random variable $\beta_1 \sim Beta(1, \alpha)$. By the definition of the <a href="http://en.wikipedia.org/wiki/Beta_distribution">Beta distribution</a>, this will be a real number between 0 and 1, with expected value $1 / (1 + \alpha)$. Break off the stick at $\beta_1$; $w_1$ is then the length of the stick on the left.</li>
<li>Now take the stick to the right, and generate $\beta_2 \sim Beta(1, \alpha)$. Break off the stick $\beta_2$ into the stick. Again, $w_2$ is the length of the stick to the left, i.e., $w_2 = (1 - \beta_1) \beta_2$.</li>
<li>And so on.</li>
</ul>
<p>Thus, the Stick-Breaking process is simply the CRP or Polya Urn Model from a different point of view. For example, assigning customers to table 1 according to the Chinese Restaurant Process is equivalent to assigning customers to table 1 with probability $w_1$.</p>
<h3>Code</h3>
<p>Here’s some R code for simulating a Stick-Breaking process:</p>
<figcaption><span>Stick-Breaking Process </span><a href="https://github.com/echen/dirichlet-process/blob/master/stick_breaking_process.R">stick_breaking_process.R</a></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
</pre></td><td class="code"><pre><code class="r"><span class="line"><span class="c1"># Return a vector of weights drawn from a stick-breaking process</span>
</span><span class="line"><span class="c1"># with dispersion `alpha`.</span>
</span><span class="line"><span class="c1">#</span>
</span><span class="line"><span class="c1"># Recall that the kth weight is</span>
</span><span class="line"><span class="c1"># \beta_k = (1 - \beta_1) * (1 - \beta_2) * ... * (1 - \beta_{k-1}) * beta_k</span>
</span><span class="line"><span class="c1"># where each $\\beta\_i$ is drawn from a Beta distribution</span>
</span><span class="line"><span class="c1"># \beta_i ~ Beta(1, \alpha)</span>
</span><span class="line">stick_breaking_process <span class="o">=</span> <span class="kr">function</span><span class="p">(</span>num_weights<span class="p">,</span> alpha<span class="p">)</span> <span class="p">{</span>
</span><span class="line"> betas <span class="o">=</span> rbeta<span class="p">(</span>num_weights<span class="p">,</span> <span class="m">1</span><span class="p">,</span> alpha<span class="p">)</span>
</span><span class="line"> remaining_stick_lengths <span class="o">=</span> c<span class="p">(</span><span class="m">1</span><span class="p">,</span> cumprod<span class="p">(</span><span class="m">1</span> <span class="o">-</span> betas<span class="p">))[</span><span class="m">1</span>:num_weights<span class="p">]</span>
</span><span class="line"> weights <span class="o">=</span> remaining_stick_lengths <span class="o">*</span> betas
</span><span class="line"> weights
</span><span class="line"><span class="p">}</span>
</span></code></pre></td></tr></table></div>
<p>And here’s some sample output:</p>
<p><a href="http://i.imgur.com/8gzowzf.png"><img src="http://i.imgur.com/8gzowzf.png" alt="Stick-Breaking Process, alpha = 1" /></a></p>
<p><a href="http://i.imgur.com/yM0Wckw.png"><img src="http://i.imgur.com/yM0Wckw.png" alt="Stick-Breaking Process, alpha = 3" /></a></p>
<p><a href="http://i.imgur.com/8tATMw6.png"><img src="http://i.imgur.com/8tATMw6.png" alt="Stick-Breaking Process, alpha = 5" /></a></p>
<p>Notice that for low values of alpha, the stick weights are concentrated on the first few weights (meaning our data points are concentrated on a few clusters), while the weights become more evenly dispersed as we increase alpha (meaning we posit more clusters in our data points).</p>
<h2>Dirichlet Process</h2>
<p>Suppose we run a Polya Urn Model several times, where we sample colors from a base distribution $G_0$. Each run produces a distribution of colors in the urn (say, 5% blue balls, 3% red balls, 2% pink balls, etc.), and the distribution will be different each time (for example, 5% blue balls in run 1, but 1% blue balls in run 2).</p>
<p>For example, let’s look again at the plots from above, where I generated samples from a Polya Urn Model with the standard unit normal as the base distribution:</p>
<p><a href="http://i.imgur.com/4aOd7TI.png"><img src="http://i.imgur.com/4aOd7TI.png" alt="Polya Urn Model, Alpha = 1" /></a></p>
<p><a href="http://i.imgur.com/vTLeppf.png"><img src="http://i.imgur.com/vTLeppf.png" alt="Polya Urn Model, Alpha = 5" /></a></p>
<p><a href="http://i.imgur.com/SQPZY95.png"><img src="http://i.imgur.com/SQPZY95.png" alt="Polya Urn Model, Alpha = 25" /></a></p>
<p><a href="http://i.imgur.com/PFmE0q3.png"><img src="http://i.imgur.com/PFmE0q3.png" alt="Polya Urn Model, Alpha = 50" /></a></p>
<p>Each run of the Polya Urn Model produces a slighly different distribution, though each is “centered” in some fashion around the standard Gaussian I used as base. In other words, the Polya Urn Model gives us a <strong>distribution over distributions</strong> (we get a distribution of ball colors, and this distribution of colors changes each time) – and so we finally get to the Dirichlet Process.</p>
<p>Formally, given a base distribution $G_0$ and a dispersion parameter $\alpha$, a sample from the Dirichlet Process $DP(G_0, \alpha)$ is a distribution $G \sim DP(G_0, \alpha)$. This sample $G$ can be thought of as a distribution of colors in a single simulation of the Polya Urn Model; sampling from $G$ gives us the balls in the urn.</p>
<p>So here’s the connection between the Chinese Restaurant Process, the Polya Urn Model, the Stick-Breaking Process, and the Dirichlet Process:</p>
<ul>
<li><strong>Dirichlet Process</strong>: Suppose we want samples $x_i \sim G$, where $G$ is a distribution sampled from the Dirichlet Process $G \sim DP(G_0, \alpha)$.</li>
<li><strong>Polya Urn Model</strong>: One way to generate these values $x_i$ would be to take a Polya Urn Model with color distribution $G_0$ and dispersion $\alpha$. ($x_i$ would be the color of the ith ball in the urn.)</li>
<li><strong>Chinese Restaurant Process</strong>: Another way to generate $x_i$ would be to first assign tables to customers according to a Chinese Restaurant Process with dispersion $\alpha$. Every customer at the nth table would then be given the same value (color) sampled from $G_0$. ($x_i$ would be the value given to the ith customer; $x_i$ can also be thought of as the food at table $i$, or as the parameters of table $i$.)</li>
<li><strong>Stick-Breaking Process</strong>: Finally, we could generate weights $w_k$ according to a Stick-Breaking Process with dispersion $\alpha$. Next, we would give each weight $w_k$ a value (or color) $v_k$ sampled from $G_0$. Finally, we would assign $x_i$ to value (color) $v_k$ with probability $w_k$.</li>
</ul>
<h1>Recap</h1>
<p>Let’s summarize what we’ve discussed so far.</p>
<p>We have a bunch of data points $p_i$ that we want to cluster, and we’ve described four essentially equivalent generative models that allow us to describe how each cluster and point could have arisen.</p>
<p>In the <strong>Chinese Restaurant Process</strong>:</p>
<ul>
<li>We generate table assignments $g_1, \ldots, g_n \sim CRP(\alpha)$ according to a Chinese Restaurant Process. ($g_i$ is the table assigned to datapoint $i$.)</li>
<li>We generate table parameters $\phi_1, \ldots, \phi_m \sim G_0$ according to the base distribution $G_0$, where $\phi_k$ is the parameter for the kth distinct group.</li>
<li>Given table assignments and table parameters, we generate each datapoint $p_i \sim F(\phi_{g_i})$ from a distribution $F$ with the specified table parameters. (For example, $F$ could be a Gaussian, and $\phi_i$ could be a parameter vector specifying the mean and standard deviation).</li>
</ul>
<p>In the <strong>Polya Urn Model</strong>:</p>
<ul>
<li>We generate colors $\phi_1, \ldots, \phi_n \sim Polya(G_0, \alpha)$ according to a Polya Urn Model. ($\phi_i$ is the color of the ith ball.)</li>
<li>Given ball colors, we generate each datapoint $p_i \sim F(\phi_i)$.</li>
</ul>
<p>In the <strong>Stick-Breaking Process</strong>:</p>
<ul>
<li>We generate group probabilities (stick lengths) $w_1, \ldots, w_{\infty} \sim Stick(\alpha)$ according to a Stick-Breaking process.</li>
<li>We generate group parameters $\phi_1, \ldots, \phi_{\infty} \sim G_0$ from $G_0$, where $\phi_k$ is the parameter for the kth distinct group.</li>
<li>We generate group assignments $g_1, \ldots, g_n \sim Multinomial(w_1, \ldots, w_{\infty})$ for each datapoint.</li>
<li>Given group assignments and group parameters, we generate each datapoint $p_i \sim F(\phi_{g_i})$.</li>
</ul>
<p>In the <strong>Dirichlet Process</strong>:</p>
<ul>
<li>We generate a distribution $G \sim DP(G_0, \alpha)$ from a Dirichlet Process with base distribution $G_0$ and dispersion parameter $\alpha$.</li>
<li>We generate group-level parameters $x_i \sim G$ from $G$, where $x_i$ is the group parameter for the ith datapoint. (Note: this is not the same as $\phi_i$. $x_i$ is the parameter associated to the group that the ith datapoint belongs to, whereas $\phi_k$ is the parameter of the kth distinct group.)</li>
<li>Given group-level parameters $x_i$, we generate each datapoint $p_i \sim F(x_i)$.</li>
</ul>
<p>Also, remember that each model naturally allows the number of clusters to grow as more points come in.</p>
<h1>Inference in the Dirichlet Process Mixture</h1>
<p>So we’ve described a generative model that allows us to calculate the probability of any particular set of group assignments to data points, but we haven’t described how to actually learn a good set of group assignments.</p>
<p>Let’s briefly do this now. Very roughly, the <strong>Gibbs sampling</strong> approach works as follows:</p>
<ul>
<li>Take the set of data points, and randomly initialize group assignments.</li>
<li>Pick a point. Fix the group assignments of all the other points, and assign the chosen point a new group (which can be either an existing cluster or a new cluster) with a CRP-ish probability (as described in the models above) that depends on the group assignments and values of all the other points.</li>
<li>We will eventually converge on a good set of group assignments, so repeat the previous step until happy.</li>
</ul>
<p>For more details, <a href="http://www.cs.toronto.edu/~radford/ftp/mixmc.pdf">this paper</a> provides a good description. Philip Resnick and Eric Hardisty also have a friendlier, more general description of Gibbs sampling (plus an application to naive Bayes) <a href="http://www.cs.umd.edu/~hardisty/papers/gsfu.pdf">here</a>.</p>
<h1>Fast Food Application: Clustering the McDonald’s Menu</h1>
<p>Finally, let’s show an application of the Dirichlet Process Mixture. Unfortunately, I didn’t have a data set of people’s food habits offhand, so instead I took <a href="http://nutrition.mcdonalds.com/nutritionexchange/nutritionfacts.pdf">this list</a> of McDonald’s foods and nutrition facts.</p>
<p>After normalizing each item to have an equal number of calories, and representing each item as a vector of <strong>(total fat, cholesterol, sodium, dietary fiber, sugars, protein, vitamin A, vitamin C, calcium, iron, calories from fat, satured fat, trans fat, carbohydrates)</strong>, I ran <a href="http://scikit-learn.sourceforge.net/dev/index.html">scikit-learn</a>’s <a href="http://scikit-learn.sourceforge.net/dev/modules/mixture.html">Dirichlet Process Gaussian Mixture Model</a> to cluster McDonald’s menu based on nutritional value.</p>
<p>First, how does the number of clusters inferred by the Dirichlet Process mixture vary as we feed in more (randomly ordered) points?</p>
<p><a href="http://i.imgur.com/36Pxuq3.png"><img src="http://i.imgur.com/36Pxuq3.png" alt="Growth of Number of Clusters" /></a></p>
<p>As expected, the Dirichlet Process model discovers more and more clusters as more and more food items arrive. (And indeed, the number of clusters appears to grow logarithmically, which can in fact be proved.)</p>
<p>How many clusters does the mixture model infer from the entire dataset? Running the Gibbs sampler several times, we find that the number of clusters tends around 11:</p>
<p><a href="http://i.imgur.com/qbkq1uo.png"><img src="http://i.imgur.com/qbkq1uo.png" alt="Number of clusters" /></a></p>
<p>Let’s dive into one of these clusterings.</p>
<p><strong>Cluster 1 (Desserts)</strong></p>
<p>Looking at a sample of foods from the first cluster, we find a lot of desserts and dessert-y drinks:</p>
<ul>
<li>Caramel Mocha</li>
<li>Frappe Caramel</li>
<li>Iced Hazelnut Latte</li>
<li>Iced Coffee</li>
<li>Strawberry Triple Thick Shake</li>
<li>Snack Size McFlurry</li>
<li>Hot Caramel Sundae</li>
<li>Baked Hot Apple Pie</li>
<li>Cinnamon Melts</li>
<li>Kiddie Cone</li>
<li>Strawberry Sundae</li>
</ul>
<p>We can also look at the nutritional profile of some foods from this cluster (after <a href="http://en.wikipedia.org/wiki/Standard_score">z-scaling</a> each nutrition dimension to have mean 0 and standard deviation 1):</p>
<p><a href="http://i.imgur.com/oM1aPCE.png"><img src="http://i.imgur.com/oM1aPCE.png" alt="Cluster 1" /></a></p>
<p>We see that foods in this cluster tend to be high in trans fat and low in vitamins, protein, fiber, and sodium.</p>
<p><strong>Cluster 2 (Sauces)</strong></p>
<p>Here’s a sample from the second cluster, which contains a lot of sauces:</p>
<ul>
<li>Hot Mustard Sauce</li>
<li>Spicy Buffalo Sauce</li>
<li>Newman’s Own Low Fat Balsamic Vinaigrette</li>
</ul>
<p>And looking at the nutritional profile of points in this cluster, we see that it’s heavy in sodium and fat:</p>
<p><a href="http://i.imgur.com/3bOYEBz.png"><img src="http://i.imgur.com/3bOYEBz.png" alt="Cluster 2" /></a></p>
<p><strong>Cluster 3 (Burgers, Crispy Foods, High-Cholesterol)</strong></p>
<p>The third cluster is very burgery:</p>
<ul>
<li>Hamburger</li>
<li>Cheeseburger</li>
<li>Filet-O-Fish</li>
<li>Quarter Pounder with Cheese</li>
<li>Premium Grilled Chicken Club Sandwich</li>
<li>Ranch Snack Wrap</li>
<li>Premium Asian Salad with Crispy Chicken</li>
<li>Butter Garlic Croutons</li>
<li>Sausage McMuffin</li>
<li>Sausage McGriddles</li>
</ul>
<p>It’s also high in fat and sodium, and low in carbs and sugar</p>
<p><a href="http://i.imgur.com/XdoVfXG.png"><img src="http://i.imgur.com/XdoVfXG.png" alt="Cluster 3" /></a></p>
<p><strong>Cluster 4 (Creamy Sauces)</strong></p>
<p>Interestingly, even though we already found a cluster of sauces above, we discover another one as well. These sauces appear to be much more cream-based:</p>
<ul>
<li>Creamy Ranch Sauce</li>
<li>Newman’s Own Creamy Caesar Dressing</li>
<li>Coffee Cream</li>
<li>Iced Coffee with Sugar Free Vanilla Syrup</li>
</ul>
<p>Nutritionally, these sauces are higher in calories from fat, and much lower in sodium:</p>
<p><a href="http://i.imgur.com/4gQzyLO.png"><img src="http://i.imgur.com/4gQzyLO.png" alt="Cluster 4" /></a></p>
<p><strong>Cluster 5 (Salads)</strong></p>
<p>Here’s a salad cluster. A lot of salads also appeared in the third cluster (along with hamburgers and McMuffins), but that’s because those salads also all contained crispy chicken. The salads in this cluster are either crisp-free or have their chicken grilled instead:</p>
<ul>
<li>Premium Southwest Salad with Grilled Chicken</li>
<li>Premium Caesar Salad with Grilled Chicken</li>
<li>Side Salad</li>
<li>Premium Asian Salad without Chicken</li>
<li>Premium Bacon Ranch Salad without Chicken</li>
</ul>
<p>This is reflected in the higher content of iron, vitamin A, and fiber:</p>
<p><a href="http://i.imgur.com/hqkpKz5.png"><img src="http://i.imgur.com/hqkpKz5.pngg" alt="Cluster 5" /></a></p>
<p><strong>Cluster 6 (More Sauces)</strong></p>
<p>Again, we find another cluster of sauces:</p>
<ul>
<li>Ketchup Packet</li>
<li>Barbeque Sauce</li>
<li>Chipotle Barbeque Sauce</li>
</ul>
<p>These are still high in sodium, but much lower in fat compared to the other sauce clusters:</p>
<p><a href="http://i.imgur.com/i2Y2M6u.png"><img src="http://i.imgur.com/i2Y2M6u.png" alt="Cluster 6" /></a></p>
<p><strong>Cluster 7 (Fruit and Maple Oatmeal)</strong></p>
<p>Amusingly, fruit and maple oatmeal is in a cluster by itself:</p>
<ul>
<li>Fruit & Maple Oatmeal</li>
</ul>
<p><a href="http://i.imgur.com/qoyNFK4.png"><img src="http://i.imgur.com/qoyNFK4.png" alt="Cluster 7" /></a></p>
<p><strong>Cluster 8 (Sugary Drinks)</strong></p>
<p>We also get a cluster of sugary drinks:</p>
<ul>
<li>Strawberry Banana Smoothie</li>
<li>Wild Berry Smoothie</li>
<li>Iced Nonfat Vanilla Latte</li>
<li>Nonfat Hazelnut</li>
<li>Nonfat Vanilla Cappuccino</li>
<li>Nonfat Caramel Cappuccino</li>
<li>Sweet Tea</li>
<li>Frozen Strawberry Lemonade</li>
<li>Coca-Cola</li>
<li>Minute Maid Orange Juice</li>
</ul>
<p>In addition to high sugar content, this cluster is also high in carbohydrates and calcium, and low in fat.</p>
<p><a href="http://i.imgur.com/MNdKmrr.png"><img src="http://i.imgur.com/MNdKmrr.png" alt="Cluster 8" /></a></p>
<p><strong>Cluster 9 (Breakfast Foods)</strong></p>
<p>Here’s a cluster of high-cholesterol breakfast foods:</p>
<ul>
<li>Sausage McMuffin with Egg</li>
<li>Sausage Burrito</li>
<li>Egg McMuffin</li>
<li>Bacon, Egg & Cheese Biscuit</li>
<li>McSkillet Burrito with Sausage</li>
<li>Big Breakfast with Hotcakes</li>
</ul>
<p><a href="http://i.imgur.com/m67oVsY.png"><img src="http://i.imgur.com/m67oVsY.png" alt="Cluster 9" /></a></p>
<p><strong>Cluster 10 (Coffee Drinks)</strong></p>
<p>We find a group of coffee drinks next:</p>
<ul>
<li>Nonfat Cappuccino</li>
<li>Nonfat Latte</li>
<li>Nonfat Latte with Sugar Free Vanilla Syrup</li>
<li>Iced Nonfat Latte</li>
</ul>
<p>These are much higher in calcium and protein, and lower in sugar, than the other drink cluster above:</p>
<p><a href="http://i.imgur.com/UB8i5j4.png"><img src="http://i.imgur.com/UB8i5j4.png" alt="Cluster 11" /></a></p>
<p><strong>Cluster 11 (Apples)</strong></p>
<p>Here’s a cluster of apples:</p>
<ul>
<li>Apple Dippers with Low Fat Caramel Dip</li>
<li>Apple Slices</li>
</ul>
<p>Vitamin C, check.</p>
<p><a href="http://i.imgur.com/w6VBaBt.png"><img src="http://i.imgur.com/w6VBaBt.png" alt="Cluster 10" /></a></p>
<p>And finally, here’s an overview of all the clusters at once (using a different clustering run):</p>
<p><a href="http://i.imgur.com/RJ6FuGH.png"><img src="http://i.imgur.com/RJ6FuGH.png" alt="All Clusters" /></a></p>
<h1>No More!</h1>
<p>I’ll end with a couple notes:</p>
<ul>
<li>Kevin Knight has a <a href="http://www.isi.edu/natural-language/people/bayes-with-tears.pdf">hilarious introduction</a> to Bayesian inference that describes some applications of nonparametric Bayesian techniques to computational linguistics (though I don’t think he ever quite says “nonparametric Bayes” directly).</li>
<li>In the Chinese Restaurant Process, each customer sits at a single table. The <a href="http://en.wikipedia.org/wiki/Chinese_restaurant_process#The_Indian_buffet_process">Indian Buffet Process</a> is an extension that allows customers to sample food from multiple tables (i.e., belong to multiple clusters).</li>
<li>The Chinese Restaurant Process, the Polya Urn Model, and the Stick-Breaking Process are all <em>sequential</em> models for generating groups: to figure out table parameters in the CRP, for example, you wait for customer 1 to come in, then customer 2, then customer 3, and so on. The equivalent Dirichlet Process, on the other hand, is a <em>parallel</em> model for generating groups: just sample $G \sim DP(G_0, alpha)$, and then all your group parameters can be independently generated by sampling from $G$ at once. This duality is an instance of a more general phenomenon known as <a href="http://en.wikipedia.org/wiki/De_Finetti's_theorem">de Finetti’s theorem</a>.</li>
</ul>
<p>And that’s it.</p>
</div>
Instant Interactive Visualization with d3 + ggplot22012-03-05T04:15:00+01:002012-03-05T04:15:00+01:00Edwin Chentag:blog.echen.me,2012-03-05:/2012/03/05/instant-interactive-visualization-with-d3-and-ggplot2/
<div class="entry-content"><p>It’s often easier to understand a chart than a table. So why is it still so hard to make a simple data graphic, and why am I still bombarded by mind-numbing reams of raw <em>numbers</em>?</p>
<p>(Yeah, I love <a href="http://blog.echen.me/2012/01/17/quick-introduction-to-ggplot2/">ggplot2</a> to death. But sometimes I want a little more interaction …</p></div>
<div class="entry-content"><p>It’s often easier to understand a chart than a table. So why is it still so hard to make a simple data graphic, and why am I still bombarded by mind-numbing reams of raw <em>numbers</em>?</p>
<p>(Yeah, I love <a href="http://blog.echen.me/2012/01/17/quick-introduction-to-ggplot2/">ggplot2</a> to death. But sometimes I want a little more interaction, and sometimes all I want is to drag-and-drop and be done.)</p>
<p>So I’ve been experimenting with <a href="http://minifolds.herokuapp.com/graphs/1?x=health&y=speed&size=intelligence&color=age&group=height">a small, ggplot2-inspired d3 app</a>.</p>
<p>Simply drop a file, and bam! Instant scatterplot:</p>
<p><a href="http://minifolds.herokuapp.com/graphs/1?x=health&y=speed"><img src="http://i.imgur.com/Dakasn5.png" alt="Swiss Roll B&W" /></a></p>
<p>But wait – that’s only 2 dimensions. You can add some more through color, size, and groups:</p>
<p><a href="http://minifolds.herokuapp.com/graphs/1?x=health&y=speed&size=intelligence&color=age&group=height"><img src="http://i.imgur.com/f6iCgHw.png" alt="Swiss Roll Edit" /></a></p>
<p>(Click <a href="http://minifolds.herokuapp.com/graphs/1?x=health&y=speed&size=intelligence&color=age&group=height">here</a> to play with the data yourself.)</p>
<p>And you can easily switch which variables are getting plotted, and see all the information associated with each point.</p>
<p><a href="http://minifolds.herokuapp.com/graphs/1?x=weight&y=speed&size=health&color=age&group=height"><img src="http://i.imgur.com/7W7OZwZ.png" alt="Swiss Roll Pivot" /></a></p>
<p>(Same dataset, different aesthetic assignments.)</p>
<p>I’m thinking of adding more kinds of charts, support for categorical variables, more interactivity (sliders to interact with other dimensions?!), and making the UI even easier (e.g., simplify column naming). In the meantime, the code is <a href="https://github.com/echen/minifolds">here</a> on Github, and tips and suggestions are welcome!</p>
</div>
Movie Recommendations and More via MapReduce and Scalding2012-02-09T04:15:00+01:002012-02-09T04:15:00+01:00Edwin Chentag:blog.echen.me,2012-02-09:/2012/02/09/movie-recommendations-and-more-via-mapreduce-and-scalding/
<div class="entry-content"><p><em>Scalding is an in-house MapReduce framework that Twitter recently open-sourced. Like <a href="http://pig.apache.org/">Pig</a>, it provides an abstraction on top of MapReduce that makes it easy to write big data jobs in a syntax that’s simple and concise. Unlike Pig, Scalding is written in pure Scala – which means all the power …</em></p></div>
<div class="entry-content"><p><em>Scalding is an in-house MapReduce framework that Twitter recently open-sourced. Like <a href="http://pig.apache.org/">Pig</a>, it provides an abstraction on top of MapReduce that makes it easy to write big data jobs in a syntax that’s simple and concise. Unlike Pig, Scalding is written in pure Scala – which means all the power of Scala and the JVM is already built-in. No more UDFs, folks!</em></p>
<p>This is going to be an in-your-face introduction to <a href="https://github.com/twitter/scalding">Scalding</a>, Twitter’s (Scala + Cascading) MapReduce framework.</p>
<p>In 140: instead of forcing you to write raw <code>map</code> and <code>reduce</code> functions, Scalding allows you to write <em>natural</em> code like</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="Example.scala" data-gist-hide-footer="true"></code>
<p>Not much different from the Ruby you’d write to compute tweet distributions over <em>small</em> data? <strong>Exactly.</strong></p>
<p>Two notes before we begin:</p>
<ul>
<li><a href="https://github.com/echen/scaldingale">This Github repository</a> contains all the code used.</li>
<li>For a gentler introduction to Scalding, see <a href="https://github.com/twitter/scalding/wiki/Getting-Started">this Getting Started guide</a> on the Scalding wiki.</li>
</ul>
<h1>Movie Similarities</h1>
<p>Imagine you run an online movie business, and you want to generate movie recommendations. You have a rating system (people can rate movies with 1 to 5 stars), and we’ll assume for simplicity that all of the ratings are stored in a TSV file somewhere.</p>
<p>Let’s start by reading the ratings into a Scalding job.</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="MovieSimilarities1.scala" data-gist-hide-footer="true"></code>
<p>You want to calculate how similar pairs of movies are, so that if someone watches <em>The Lion King</em>, you can recommend films like <em>Toy Story</em>. So how should you define the similarity between two movies?</p>
<p>One way is to use their <strong>correlation</strong>:</p>
<ul>
<li>For every pair of movies A and B, find all the people who rated both A and B.</li>
<li>Use these ratings to form a Movie A vector and a Movie B vector.</li>
<li>Calculate the correlation between these two vectors.</li>
<li>Whenever someone watches a movie, you can then recommend the movies most correlated with it.</li>
</ul>
<p>Let’s start with the first two steps.</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="MovieSimilarities2.scala" data-gist-hide-footer="true"></code>
<p>Before using these rating pairs to calculate correlation, let’s stop for a bit.</p>
<p>Since we’re explicitly thinking of movies as <strong>vectors</strong> of ratings, it’s natural to compute some very vector-y things like norms and dot products, as well as the length of each vector and the sum over all elements in each vector. So let’s compute these:</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="MovieSimilarities3.scala" data-gist-hide-footer="true"></code>
<p>To summarize, each row in <code>vectorCalcs</code> now contains the following fields:</p>
<ul>
<li><strong>movie, movie2</strong></li>
<li><strong>numRaters, numRaters2</strong>: the total number of people who rated each movie</li>
<li><strong>size</strong>: the number of people who rated both movie and movie2</li>
<li><strong>dotProduct</strong>: dot product between the movie vector (a vector of ratings) and the movie2 vector (also a vector of ratings)</li>
<li><strong>ratingSum, rating2sum</strong>: sum over all elements in each ratings vector</li>
<li><strong>ratingNormSq, rating2Normsq</strong>: squared norm of each vector</li>
</ul>
<p>So let’s go back to calculating the correlation between movie and movie2. We could, of course, calculate correlation in the standard way: find the covariance between the movie and movie2 ratings, and divide by their standard deviations.</p>
<p>But recall that we can also write correlation in the following form:</p>
<p>$Corr(X, Y) = \frac{n \sum xy - \sum x \sum y}{\sqrt{n \sum x^2 - (\sum x)^2} \sqrt{n \sum y^2 - (\sum y)^2}}$</p>
<p>(See the <a href="http://en.wikipedia.org/wiki/Correlation_and_dependence">Wikipedia page</a> on correlation.)</p>
<p>Notice that every one of the elements in this formula is a field in <code>vectorCalcs</code>! So instead of using the standard calculation, we can use this form instead:</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="MovieSimilarities4.scala" data-gist-hide-footer="true"></code>
<p>And that’s it! To see the full code, check out the Github repository <a href="https://github.com/echen/scaldingale">here</a>.</p>
<h1>Book Similarities</h1>
<p>Let’s run this code over some real data. Unfortunately, I didn’t have a clean source of movie ratings available, so instead I used <a href="http://www.informatik.uni-freiburg.de/~cziegler/BX/">this dataset</a> of 1 million book ratings.</p>
<p>I ran a quick command, using the handy <a href="https://github.com/twitter/scalding/wiki/Scald.rb">scald.rb script</a> that Scalding provides…</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="scald.rb" data-gist-hide-footer="true"></code>
<p>…and here’s a sample of the top output I got:</p>
<p><a href="http://i.imgur.com/e6wpQOt.png"><img src="http://i.imgur.com/e6wpQOt.png" alt="Top Book-Crossing Pairs" /></a></p>
<p>As we’d expect, we see that</p>
<ul>
<li><em>Harry Potter</em> books are similar to other <em>Harry Potter</em> books</li>
<li><em>Lord of the Rings</em> books are similar to other <em>Lord of the Rings</em> books</li>
<li>Tom Clancy is similar to John Grisham</li>
<li>Chick lit (<em>Summer Sisters</em>, by Judy Blume) is similar to chick lit (<em>Bridget Jones</em>)</li>
</ul>
<p>Just for fun, let’s also look at books similar to <em>The Great Gatsby</em>:</p>
<p><a href="http://i.imgur.com/FFzySua.png"><img src="http://i.imgur.com/FFzySua.png" alt="Great Gatsby" /></a></p>
<p>(Schoolboy memories, exactly.)</p>
<h1>More Similarity Measures</h1>
<p>Of course, there are lots of other similarity measures we could use besides correlation.</p>
<h2>Cosine Similarity</h2>
<p><a href="http://en.wikipedia.org/wiki/Cosine_similarity">Cosine similarity</a> is a another common vector-based similarity measure.</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="MovieSimilarities5.scala" data-gist-hide-footer="true"></code>
<h2>Correlation, Take II</h2>
<p>We can also also add a <em>regularized</em> correlation, by (say) adding N virtual movie pairs that have zero correlation. This helps avoid noise if some movie pairs have very few raters in common (for example, <em>The Great Gatsby</em> had an unlikely raw correlation of 1 with many other books, due simply to the fact that those book pairs had very few ratings).</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="MovieSimilarities6.scala" data-gist-hide-footer="true"></code>
<h2>Jaccard Similarity</h2>
<p>Recall that <a href="http://blog.echen.me/blog/2011/10/24/winning-the-netflix-prize-a-summary/">one of the lessons of the Netflix prize</a> was that implicit data can be quite useful – the mere fact that you rate a James Bond movie, even if you rate it quite horribly, suggests that you’d probably be interested in similar action films. So we can also ignore the value itself of each rating and use a <em>set</em>-based similarity measure like <a href="http://en.wikipedia.org/wiki/Jaccard_index">Jaccard similarity</a>.</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="MovieSimilarities7.scala" data-gist-hide-footer="true"></code>
<h2>Incorporation</h2>
<p>Finally, let’s add all these similarity measures to our output.</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="MovieSimilarities8.scala" data-gist-hide-footer="true"></code>
<h1>Book Similarities Revisited</h1>
<p>Let’s take another look at the book similarities above, now that we have these new fields.</p>
<p>Here are some of the top Book-Crossing pairs, sorted by their shrunk correlation:</p>
<p><a href="http://i.imgur.com/aR4xIBL.png"><img src="http://i.imgur.com/aR4xIBL.png" alt="Top Book-Crossing Pairs" /></a></p>
<p>Notice how regularization affects things: the <em>Dark Tower</em> pair has a pretty high raw correlation, but relatively few ratings (reducing our confidence in the raw correlation), so it ends up below the others.</p>
<p>And here are books similar to <em>The Great Gatsby</em>, this time ordered by cosine similarity:</p>
<p><a href="http://i.imgur.com/lzZdXU5.png"><img src="http://i.imgur.com/lzZdXU5.png" alt="Great Gatsby" /></a></p>
<h1>Input Abstraction</h1>
<p>So our code right now is tied to our specific <code>ratings.tsv</code> input. But what if we change the way we store our ratings, or what if we want to generate similarities for something entirely different?</p>
<p>Let’s abstract away our input. We’ll create a <a href="https://github.com/echen/scaldingale/blob/master/VectorSimilarities.scala">VectorSimilarities class</a> that represents input data in the following format:</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="VectorSimilarities.scala" data-gist-hide-footer="true"></code>
<p>Whenever we want to define a new input format, we simply subclass <code>VectorSimilarities</code> and provide a concrete implementation of the <code>input</code> method.</p>
<h2>Book-Crossings</h2>
<p>For example, here’s a class I could have used to generate the book recommendations above:</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="BookCrossing.scala" data-gist-hide-footer="true"></code>
<p>The input method simply reads from a TSV file and lets the <code>VectorSimilarities</code> superclass do all the work. Instant recommendations, BOOM.</p>
<h2>Song Similarities with Twitter + iTunes</h2>
<p>But why limit ourselves to books? We do, after all, have Twitter at our fingertips…</p>
<blockquote class="twitter-tweet"><p>rated Born This Way by Lady GaGa 5 stars <a href="http://t.co/wTYAwWqm" title="http://itun.es/iSg92N">itun.es/iSg92N</a> <a href="https://twitter.com/search/%2523iTunes">#iTunes</a></p>— gggf (@GalMusic92) <a href="https://twitter.com/GalMusic92/status/167267017865428996" data-datetime="2012-02-08T15:22:19+00:00">February 8, 2012</a></blockquote>
<script src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Since iTunes lets you send a tweet whenever you rate a song, we can use these to generate music recommendations!</p>
<p>Again, we create a new class that overrides the abstract <code>input</code> defined in <code>VectorSimilarities</code>…</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="iTunes.scala" data-gist-hide-footer="true"></code>
<p>…and snap! Here are some songs you might like if you recently listened to <strong>Beyoncé</strong>:</p>
<p><a href="http://i.imgur.com/8E8q4y1.png"><img src="http://i.imgur.com/8E8q4y1.png" alt="Jason Mraz" /></a></p>
<p>And some recommended songs if you like <strong>Lady Gaga</strong>:</p>
<p><a href="http://i.imgur.com/AUPR87F.png"><img src="http://i.imgur.com/AUPR87F.png" alt="Lady Gaga" /></a></p>
<p>GG Pandora.</p>
<h2>Location Similarities with Foursquare Check-ins</h2>
<p>But what if we don’t have explicit ratings? For example, we could be a news site that wants to generate article recommendations, and maybe we only have user <em>visits</em> on each story.</p>
<p>Or what if we want to generate restaurant or tourist recommendations, when all we know is who visits each location?</p>
<blockquote class="twitter-tweet"><p>I’m at Empire State Building (350 5th Ave., btwn 33rd & 34th St., New York) <a href="http://t.co/q6tXzf3n" title="http://4sq.com/zZ5xGd">4sq.com/zZ5xGd</a></p>— Simon Ackerman (@SimonAckerman) <a href="https://twitter.com/SimonAckerman/status/167232054247956481" data-datetime="2012-02-08T13:03:23+00:00">February 8, 2012</a></blockquote>
<script src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Let’s finally make Foursquare check-ins useful. (I kid, I kid.)</p>
<p>Instead of using an explicit rating given to us, we can simply generate a dummy rating of 1 for each check-in. Correlation doesn’t make sense any more, but we can still pay attention to a measure like Jaccard simiilarity.</p>
<p>So we simply create a new class that scrapes tweets for Foursquare check-in information…</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="Foursquare.scala" data-gist-hide-footer="true"></code>
<p>…and bam! Here are locations similar to the <strong>Empire State Building</strong>:</p>
<p><a href="http://i.imgur.com/BkcbCnB.png"><img src="http://i.imgur.com/BkcbCnB.png" alt="Empire State Building" /></a></p>
<p>Here are places you might want to check out, if you check-in at <strong>Bergdorf Goodman</strong>:</p>
<p><a href="http://i.imgur.com/1IZBlUS.png"><img src="http://i.imgur.com/1IZBlUS.png" alt="Bergdorf Goodman" /></a></p>
<p>And here’s where to go after the <strong>Statue of Liberty</strong>:</p>
<p><a href="http://i.imgur.com/SktVYlM.png"><img src="http://i.imgur.com/SktVYlM.png" alt="Statue of Liberty" /></a></p>
<p>Power of Twitter, yo.</p>
<h1>RottenTomatoes Similarities</h1>
<p>UPDATE: I found some movie data after all…</p>
<blockquote class="twitter-tweet"><p>My review for ‘How to Train Your Dragon’ on Rotten Tomatoes: 4 1/2 stars ><a href="http://t.co/YTOKWLEt" title="http://bit.ly/xtw3d3">bit.ly/xtw3d3</a></p>— Benjamin West (@BenTheWest) <a href="https://twitter.com/BenTheWest/status/171772890121895936" data-datetime="2012-02-21T01:47:03+00:00">February 21, 2012</a></blockquote>
<p>So let’s use RottenTomatoes tweets to recommend movies! Here’s the code for a class that searches for RottenTomatoes tweets:</p>
<code data-gist-id="3d6bb2d5915b263f674ba9ad057ef773" data-gist-file="RottenTomatoes.scala" data-gist-hide-footer="true"></code>
<p>And here are the most similar movies discovered:</p>
<p><a href="http://i.imgur.com/QwWgZqP.png"><img src="http://i.imgur.com/QwWgZqP.png" alt="Top RottenTomatoes Movies" /></a></p>
<p>We see that</p>
<ul>
<li><em>Lord of the Rings</em>, <em>Harry Potter</em>, and <em>Star Wars</em> movies are similar to other <em>Lord of the Rings</em>, <em>Harry Potter</em>, and <em>Star Wars</em> movies</li>
<li>Big science fiction blockbusters (<em>Avatar</em>) are similar to big science fiction blockbusters (<em>Inception</em>)</li>
<li>People who like one Justin Timberlake movie (<em>Bad Teacher</em>) also like other Justin Timberlake Movies (<em>In Time</em>). Similarly with Michael Fassbender (<em>A Dangerous Method</em>, <em>Shame</em>)</li>
<li>Art house movies (<em>The Tree of Life</em>) stick together (<em>Tinker Tailor Soldier Spy</em>)</li>
</ul>
<p>Let’s also look at the movies with the most <em>negative</em> correlation:</p>
<p><a href="http://i.imgur.com/Ln7ZZrZ.png"><img src="http://i.imgur.com/Ln7ZZrZ.png" alt="Negative RottenTomatoes Movies" /></a></p>
<p>(The more you like loud and dirty popcorn movies (<em>Thor</em>) and vamp romance (<em>Twilight</em>), the less you like arthouse? SGTM.)</p>
<h1>Next Steps</h1>
<p>Hopefully I gave you a taste of the awesomeness of Scalding. To learn even more:</p>
<ul>
<li>Check out <a href="https://github.com/twitter/scalding">Scalding on Github</a>.</li>
<li>Read <a href="https://github.com/twitter/scalding/wiki/Getting-Started">this Getting Started Guide</a> on the Scalding wiki.</li>
<li>Run through <a href="https://github.com/twitter/scalding/tree/master/tutorial">this code-based introduction</a>, complete with Scalding jobs that you can run in local mode.</li>
<li>Browse <a href="https://github.com/twitter/scalding/wiki/API-Reference">the API reference</a>, which also contains many code snippets illustrating different Scalding functions (e.g., <code>map</code>, <code>filter</code>, <code>flatMap</code>, <code>groupBy</code>, <code>count</code>, <code>join</code>).</li>
<li>And all the code for this post is <a href="https://github.com/echen/scaldingale">here</a>.</li>
</ul>
<p>Watch out for more documentation soon, and you should most definitely <a href="https://twitter.com/#!/scalding">follow @Scalding</a> on Twitter for updates or to ask any questions.</p>
<h1>Mad Props</h1>
<p>And finally, a huge shoutout to <a href="https://twitter.com/argyris">Argyris Zymnis</a>, <a href="https://twitter.com/avibryant">Avi Bryant</a>, and <a href="https://twitter.com/posco">Oscar Boykin</a>, the mastermind hackers who have spent (and continue spending!) unimaginable hours making Scalding a joy to use.</p>
<p>@argyris, @avibryant, @posco: Thanks for it all. #awesomejobguys #loveit</p>
</div>
Quick Introduction to ggplot22012-01-17T04:15:00+01:002012-01-17T04:15:00+01:00Edwin Chentag:blog.echen.me,2012-01-17:/2012/01/17/quick-introduction-to-ggplot2/
<div class="entry-content"><p>This is a bare-bones introduction to <a href="http://had.co.nz/ggplot2/">ggplot2</a>, a visualization package in R. It assumes no knowledge of R.</p>
<p>For a better-looking version of this post, see <a href="https://github.com/echen/ggplot2-tutorial">this Github repository</a>, which also contains some of the <a href="https://github.com/echen/ggplot2-tutorial/tree/master/data">example datasets</a> I use and a <a href="https://github.com/echen/ggplot2-tutorial/blob/master/ggplot2-tutorial.R">literate programming version</a> of this tutorial.</p>
<h1>Preview</h1>
<p>Let’s …</p></div>
<div class="entry-content"><p>This is a bare-bones introduction to <a href="http://had.co.nz/ggplot2/">ggplot2</a>, a visualization package in R. It assumes no knowledge of R.</p>
<p>For a better-looking version of this post, see <a href="https://github.com/echen/ggplot2-tutorial">this Github repository</a>, which also contains some of the <a href="https://github.com/echen/ggplot2-tutorial/tree/master/data">example datasets</a> I use and a <a href="https://github.com/echen/ggplot2-tutorial/blob/master/ggplot2-tutorial.R">literate programming version</a> of this tutorial.</p>
<h1>Preview</h1>
<p>Let’s start with a preview of what ggplot2 can do.</p>
<p>Given Fisher’s <a href="http://en.wikipedia.org/wiki/Iris_flower_data_set">iris</a> data set and one simple command…</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">qplot<span class="p">(</span>Sepal.Length<span class="p">,</span> Petal.Length<span class="p">,</span> data <span class="o">=</span> iris<span class="p">,</span> color <span class="o">=</span> Species<span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p>…we can produce this plot of sepal length vs. petal length, colored by species.</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-specied.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-specied.png" alt="Sepal vs. Petal, Colored by Species" /></a></p>
<h1>Installation</h1>
<p>You can download R <a href="http://cran.opensourceresources.org/">here</a>. After installation, you can launch R in interactive mode by either typing <code>R</code> on the command line or opening the standard GUI (which should have been included in the download).</p>
<h1>R Basics</h1>
<h2>Vectors</h2>
<p>Vectors are a core data structure in R, and are created with <code>c()</code>. Elements in a vector must be of the same type.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">numbers <span class="o">=</span> c<span class="p">(</span><span class="m">23</span><span class="p">,</span> <span class="m">13</span><span class="p">,</span> <span class="m">5</span><span class="p">,</span> <span class="m">7</span><span class="p">,</span> <span class="m">31</span><span class="p">)</span>
</span><span class="line">names <span class="o">=</span> c<span class="p">(</span><span class="s">"edwin"</span><span class="p">,</span> <span class="s">"alice"</span><span class="p">,</span> <span class="s">"bob"</span><span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p>Elements are indexed starting at 1, and are accessed with <code>[]</code> notation.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">numbers<span class="p">[</span><span class="m">1</span><span class="p">]</span> <span class="c1"># 23</span>
</span><span class="line">names<span class="p">[</span><span class="m">1</span><span class="p">]</span> <span class="c1"># edwin</span>
</span></code></pre></td></tr></table></div>
<h2>Data frames</h2>
<p><a href="http://www.r-tutor.com/r-introduction/data-frame">Data frames</a> are like matrices, but with named columns of different types (similar to <a href="http://code.google.com/p/sqldf/">database tables</a>).</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">books <span class="o">=</span> data.frame<span class="p">(</span>
</span><span class="line"> title <span class="o">=</span> c<span class="p">(</span><span class="s">"harry potter"</span><span class="p">,</span> <span class="s">"war and peace"</span><span class="p">,</span> <span class="s">"lord of the rings"</span><span class="p">),</span> <span class="c1"># column named "title"</span>
</span><span class="line"> author <span class="o">=</span> c<span class="p">(</span><span class="s">"rowling"</span><span class="p">,</span> <span class="s">"tolstoy"</span><span class="p">,</span> <span class="s">"tolkien"</span><span class="p">),</span>
</span><span class="line"> num_pages <span class="o">=</span> c<span class="p">(</span><span class="s">"350"</span><span class="p">,</span> <span class="s">"875"</span><span class="p">,</span> <span class="s">"500"</span><span class="p">)</span>
</span><span class="line"><span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p>You can access columns of a data frame with <code>$</code>.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">books<span class="p">$</span>title <span class="c1"># c("harry potter", "war and peace", "lord of the rings")</span>
</span><span class="line">books<span class="p">$</span>author<span class="p">[</span><span class="m">1</span><span class="p">]</span> <span class="c1"># "rowling"</span>
</span></code></pre></td></tr></table></div>
<p>You can also create new columns with <code>$</code>.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">books<span class="p">$</span>num_bought_today <span class="o">=</span> c<span class="p">(</span><span class="m">10</span><span class="p">,</span> <span class="m">5</span><span class="p">,</span> <span class="m">8</span><span class="p">)</span>
</span><span class="line">books<span class="p">$</span>num_bought_yesterday <span class="o">=</span> c<span class="p">(</span><span class="m">18</span><span class="p">,</span> <span class="m">13</span><span class="p">,</span> <span class="m">20</span><span class="p">)</span>
</span><span class="line">
</span><span class="line">books<span class="p">$</span>total\_num\_bought <span class="o">=</span> books<span class="p">$</span>num_bought_today <span class="o">+</span> books<span class="p">$</span>num_bought_yesterday
</span></code></pre></td></tr></table></div>
<h2>read.table</h2>
<p>Suppose you want to import a TSV file into R as a data frame.</p>
<h3>tsv file without header</h3>
<p>For example, consider the <a href="https://github.com/echen/r-tutorial/blob/master/data/students.tsv"><code>data/students.tsv</code></a> file (with columns describing each student’s age, test score, and name).</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
</pre></td><td class="code"><pre><code class="r"><span class="line"><span class="m">13</span> <span class="m">100</span> alice
</span><span class="line"><span class="m">14</span> <span class="m">95</span> bob
</span><span class="line"><span class="m">13</span> <span class="m">82</span> eve
</span></code></pre></td></tr></table></div>
<p>We can import this file into R using <a href="http://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html"><code>read.table()</code></a>.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">students <span class="o">=</span> read.table<span class="p">(</span><span class="s">"data/students.tsv"</span><span class="p">,</span>
</span><span class="line"> header <span class="o">=</span> <span class="k-Variable">F</span><span class="p">,</span> <span class="c1"># file does not contain a header (`F` is short for `FALSE`), so we must manually specify column names </span>
</span><span class="line"> sep <span class="o">=</span> <span class="s">"\t"</span><span class="p">,</span> <span class="c1"># file is tab-delimited </span>
</span><span class="line"> col.names <span class="o">=</span> c<span class="p">(</span><span class="s">"age"</span><span class="p">,</span> <span class="s">"score"</span><span class="p">,</span> <span class="s">"name"</span><span class="p">)</span> <span class="c1"># column names</span>
</span><span class="line"><span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p>We can now access the different columns in the data frame with <code>students$age</code>, <code>students$score</code>, and <code>students$name</code>.</p>
<h3>csv file with header</h3>
<p>For an example of a file in a different format, look at the <a href="https://github.com/echen/r-tutorial/blob/master/data/studentsWithHeader.tsv"><code>data/studentsWithHeader.tsv</code></a> file.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">age<span class="p">,</span>score<span class="p">,</span>name
</span><span class="line"><span class="m">13</span><span class="p">,</span><span class="m">100</span><span class="p">,</span>alice
</span><span class="line"><span class="m">14</span><span class="p">,</span><span class="m">95</span><span class="p">,</span>bob
</span><span class="line"><span class="m">13</span><span class="p">,</span><span class="m">82</span><span class="p">,</span>eve
</span></code></pre></td></tr></table></div>
<p>Here we have the same data, but now the file is comma-delimited and contains a header. We can import this file with</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">students <span class="o">=</span> read.table<span class="p">(</span><span class="s">"data/students.tsv"</span><span class="p">,</span>
</span><span class="line"> sep <span class="o">=</span> <span class="s">","</span><span class="p">,</span>
</span><span class="line"> header <span class="o">=</span> <span class="k-Variable">T</span> <span class="c1"># first line contains column names, so we can immediately call `students$age` </span>
</span><span class="line"><span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p>(Note: there is also a <code>read.csv</code> function that uses <code>sep = ","</code> by default.)</p>
<h2>help</h2>
<p>There are many more options that <code>read.table</code> can take. For a list of these, just type <code>help(read.table)</code> (or <code>?read.table</code>) at the prompt to access documentation.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
</pre></td><td class="code"><pre><code class="r"><span class="line"><span class="c1"># These work for other functions as well.</span>
</span><span class="line">help<span class="p">(</span>read.table<span class="p">)</span>
</span><span class="line">?read.table
</span></code></pre></td></tr></table></div>
<h1>ggplot2</h1>
<p>With these R basics in place, let’s dive into the ggplot2 package.</p>
<h2>Installation</h2>
<p>One of R’s greatest strengths is its excellent set of <a href="http://cran.r-project.org/web/packages/available_packages_by_name.html">packages</a>. To install a package, you can use the <code>install.packages()</code> function.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">install.packages<span class="p">(</span><span class="s">"ggplot2"</span><span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p>To load a package into your current R session, use <code>library()</code>.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">library<span class="p">(</span>ggplot2<span class="p">)</span>
</span></code></pre></td></tr></table></div>
<h2>Scatterplots with qplot()</h2>
<p>Let’s look at how to create a scatterplot in ggplot2. We’ll use the <code>iris</code> data frame that’s automatically loaded into R.</p>
<p>What does the data frame contain? We can use the <code>head</code> function to look at the first few rows.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">head<span class="p">(</span>iris<span class="p">)</span> <span class="c1"># by default, head displays the first 6 rows. see `?head`</span>
</span><span class="line">head<span class="p">(</span>iris<span class="p">,</span> n <span class="o">=</span> <span class="m">10</span><span class="p">)</span> <span class="c1"># we can also explicitly set the number of rows to display</span>
</span><span class="line">
</span><span class="line">Sepal.Length Sepal.Width Petal.Length Petal.Width Species
</span><span class="line"> <span class="m">5.1</span> <span class="m">3.5</span> <span class="m">1.4</span> <span class="m">0.2</span> setosa
</span><span class="line"> <span class="m">4.9</span> <span class="m">3.0</span> <span class="m">1.4</span> <span class="m">0.2</span> setosa
</span><span class="line"> <span class="m">4.7</span> <span class="m">3.2</span> <span class="m">1.3</span> <span class="m">0.2</span> setosa
</span><span class="line"> <span class="m">4.6</span> <span class="m">3.1</span> <span class="m">1.5</span> <span class="m">0.2</span> setosa
</span><span class="line"> <span class="m">5.0</span> <span class="m">3.6</span> <span class="m">1.4</span> <span class="m">0.2</span> setosa
</span><span class="line"> <span class="m">5.4</span> <span class="m">3.9</span> <span class="m">1.7</span> <span class="m">0.4</span> setosa
</span></code></pre></td></tr></table></div>
<p>(The data frame actually contains three types of species: setosa, versicolor, and virginica.)</p>
<p>Let’s plot <code>Sepal.Length</code> against <code>Petal.Length</code> using ggplot2’s <code>qplot()</code> function.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">qplot<span class="p">(</span>Sepal.Length<span class="p">,</span> Petal.Length<span class="p">,</span> data <span class="o">=</span> iris<span class="p">)</span>
</span><span class="line"><span class="c1"># Plot Sepal.Length vs. Petal.Length, using data from the `iris` data frame.</span>
</span><span class="line"><span class="c1"># * First argument `Sepal.Length` goes on the x-axis.</span>
</span><span class="line"><span class="c1"># * Second argument `Petal.Length` goes on the y-axis.</span>
</span><span class="line"><span class="c1"># * `data = iris` means to look for this data in the `iris` data frame. </span>
</span></code></pre></td></tr></table></div>
<p></p>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal.png" alt="Sepal Length vs. Petal Length" /></a></p>
<p>To see where each species is located in this graph, we can color each point by adding a <code>color = Species</code> argument.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">qplot<span class="p">(</span>Sepal.Length<span class="p">,</span> Petal.Length<span class="p">,</span> data <span class="o">=</span> iris<span class="p">,</span> color <span class="o">=</span> Species<span class="p">)</span> <span class="c1"># dude!</span>
</span></code></pre></td></tr></table></div>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-specied.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-specied.png" alt="Sepal vs. Petal, Colored by Species" /></a></p>
<p>Similarly, we can let the size of each point denote sepal width, by adding a <code>size = Sepal.Width</code> argument.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">qplot<span class="p">(</span>Sepal.Length<span class="p">,</span> Petal.Length<span class="p">,</span> data <span class="o">=</span> iris<span class="p">,</span> color <span class="o">=</span> Species<span class="p">,</span> size <span class="o">=</span> Petal.Width<span class="p">)</span>
</span><span class="line"><span class="c1"># We see that Iris setosa flowers have the narrowest petals.</span>
</span></code></pre></td></tr></table></div>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-sized.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-sized.png" alt="Sepal vs. Petal, Sized by Petal Width" /></a></p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">qplot<span class="p">(</span>Sepal.Length<span class="p">,</span> Petal.Length<span class="p">,</span> data <span class="o">=</span> iris<span class="p">,</span> color <span class="o">=</span> Species<span class="p">,</span> size <span class="o">=</span> Petal.Width<span class="p">,</span> alpha <span class="o">=</span> I<span class="p">(</span><span class="m">0.7</span><span class="p">))</span>
</span><span class="line"><span class="c1"># By setting the alpha of each point to 0.7, we reduce the effects of overplotting.</span>
</span></code></pre></td></tr></table></div>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-alpha.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-alpha.png" alt="Sepal vs. Petal, with Transparency" /></a></p>
<p>Finally, let’s fix the axis labels and add a title to the plot.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">qplot<span class="p">(</span>Sepal.Length<span class="p">,</span> Petal.Length<span class="p">,</span> data <span class="o">=</span> iris<span class="p">,</span> color <span class="o">=</span> Species<span class="p">,</span>
</span><span class="line"> xlab <span class="o">=</span> <span class="s">"Sepal Length"</span><span class="p">,</span> ylab <span class="o">=</span> <span class="s">"Petal Length"</span><span class="p">,</span>
</span><span class="line"> main <span class="o">=</span> <span class="s">"Sepal vs. Petal Length in Fisher's Iris data"</span><span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-titled.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-titled.png" alt="Sepal vs. Petal, Titled" /></a></p>
<h2>Other common geoms</h2>
<p>In the scatterplot examples above, we implicitly used a <em>point</em> <strong>geom</strong>, the default when you supply two arguments to <code>qplot()</code>.</p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
</pre></td><td class="code"><pre><code class="r"><span class="line"><span class="c1"># These two invocations are equivalent.</span>
</span><span class="line">qplot<span class="p">(</span>Sepal.Length<span class="p">,</span> Petal.Length<span class="p">,</span> data <span class="o">=</span> iris<span class="p">,</span> geom <span class="o">=</span> <span class="s">"point"</span><span class="p">)</span>
</span><span class="line">qplot<span class="p">(</span>Sepal.Length<span class="p">,</span> Petal.Length<span class="p">,</span> data <span class="o">=</span> iris<span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p>But we can also easily use other types of geoms to create more kinds of plots.</p>
<h3>Barcharts: geom = “bar”</h3>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">movies <span class="o">=</span> data.frame<span class="p">(</span>
</span><span class="line"> director <span class="o">=</span> c<span class="p">(</span><span class="s">"spielberg"</span><span class="p">,</span> <span class="s">"spielberg"</span><span class="p">,</span> <span class="s">"spielberg"</span><span class="p">,</span> <span class="s">"jackson"</span><span class="p">,</span> <span class="s">"jackson"</span><span class="p">),</span>
</span><span class="line"> movie <span class="o">=</span> c<span class="p">(</span><span class="s">"jaws"</span><span class="p">,</span> <span class="s">"avatar"</span><span class="p">,</span> <span class="s">"schindler's list"</span><span class="p">,</span> <span class="s">"lotr"</span><span class="p">,</span> <span class="s">"king kong"</span><span class="p">),</span>
</span><span class="line"> minutes <span class="o">=</span> c<span class="p">(</span><span class="m">124</span><span class="p">,</span> <span class="m">163</span><span class="p">,</span> <span class="m">195</span><span class="p">,</span> <span class="m">600</span><span class="p">,</span> <span class="m">187</span><span class="p">)</span>
</span><span class="line"><span class="p">)</span>
</span><span class="line">
</span><span class="line"><span class="c1"># Plot the number of movies each director has.</span>
</span><span class="line">qplot<span class="p">(</span>director<span class="p">,</span> data <span class="o">=</span> movies<span class="p">,</span> geom <span class="o">=</span> <span class="s">"bar"</span><span class="p">,</span> ylab <span class="o">=</span> <span class="s">"# movies"</span><span class="p">)</span>
</span><span class="line"><span class="c1"># By default, the height of each bar is simply a count.</span>
</span></code></pre></td></tr></table></div>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/num-movies.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/num-movies.png" alt="# Movies" /></a></p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
</pre></td><td class="code"><pre><code class="r"><span class="line"><span class="c1"># But we can also supply a different weight.</span>
</span><span class="line"><span class="c1"># Here the height of each bar is the total running time of the director's movies.</span>
</span><span class="line">qplot<span class="p">(</span>director<span class="p">,</span> weight <span class="o">=</span> minutes<span class="p">,</span> data <span class="o">=</span> movies<span class="p">,</span> geom <span class="o">=</span> <span class="s">"bar"</span><span class="p">,</span> ylab <span class="o">=</span> <span class="s">"total length (min.)"</span><span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/total-length.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/total-length.png" alt="Total Running Time" /></a></p>
<h3>Line charts: geom = “line”</h3>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
</pre></td><td class="code"><pre><code class="r"><span class="line">qplot<span class="p">(</span>Sepal.Length<span class="p">,</span> Petal.Length<span class="p">,</span> data <span class="o">=</span> iris<span class="p">,</span> geom <span class="o">=</span> <span class="s">"line"</span><span class="p">,</span> color <span class="o">=</span> Species<span class="p">)</span>
</span><span class="line"><span class="c1"># Using a line geom doesn't really make sense here, but hey.</span>
</span></code></pre></td></tr></table></div>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-lined.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/sepal-vs-petal-lined.png" alt="Sepal vs. Petal, Lined" /></a></p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
</pre></td><td class="code"><pre><code class="r"><span class="line"><span class="c1"># `Orange` is another built-in data frame that describes the growth of orange trees.</span>
</span><span class="line">qplot<span class="p">(</span>age<span class="p">,</span> circumference<span class="p">,</span> data <span class="o">=</span> Orange<span class="p">,</span> geom <span class="o">=</span> <span class="s">"line"</span><span class="p">,</span>
</span><span class="line"> colour <span class="o">=</span> Tree<span class="p">,</span>
</span><span class="line"> main <span class="o">=</span> <span class="s">"How does orange tree circumference vary with age?"</span><span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/orange-tree-growth.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/orange-tree-growth.png" alt="Orange Tree Growth" /></a></p>
<figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
</pre></td><td class="code"><pre><code class="r"><span class="line"><span class="c1"># We can also plot both points and lines.</span>
</span><span class="line">qplot<span class="p">(</span>age<span class="p">,</span> circumference<span class="p">,</span> data <span class="o">=</span> Orange<span class="p">,</span> geom <span class="o">=</span> c<span class="p">(</span><span class="s">"point"</span><span class="p">,</span> <span class="s">"line"</span><span class="p">),</span> colour <span class="o">=</span> Tree<span class="p">)</span>
</span></code></pre></td></tr></table></div>
<p><a href="http://dl.dropbox.com/u/10506/blog/r/ggplot2/orange-tree-pointed.png"><img src="http://dl.dropbox.com/u/10506/blog/r/ggplot2/orange-tree-pointed.png" alt="Orange Tree with Points" /></a></p>
<p>And that’s it with what I’ll cover.</p>
<h1>Next Steps</h1>
<p>I skipped over a lot of aspects of R and ggplot2 in this intro.</p>
<p>For example,</p>
<ul>
<li>There are many geoms (and other functionalities) in ggplot2 that I didn’t cover, e.g., <a href="http://had.co.nz/ggplot2/geom_boxplot.html">boxplots</a> and <a href="http://had.co.nz/ggplot2/geom_histogram.html">histograms</a>.</li>
<li>I didn’t talk about ggplot2’s layering system, or the <a href="http://www.amazon.com/Grammar-Graphics-Statistics-Computing/dp/0387245448">grammar of graphics</a> it’s based on.</li>
</ul>
<p>So I’ll end with some additional resources on R and ggplot2.</p>
<ul>
<li>I don’t use it myself, but <a href="http://rstudio.org/">RStudio</a> is a popular IDE for R.</li>
<li>The <a href="http://had.co.nz/ggplot2/">official ggplot2 documentation</a> is great and has lots of examples. There’s also an excellent <a href="http://www.amazon.com/ggplot2-Elegant-Graphics-Data-Analysis/dp/0387981403">book</a>.</li>
<li><a href="http://plyr.had.co.nz/">plyr</a> is another fantastic R package that’s also by Hadley Wickham (the author of ggplot2).</li>
<li>The <a href="http://cran.r-project.org/doc/manuals/R-intro.html">official R introduction</a> is okay, but definitely not great. I haven’t found any R tutorials I really like, but I’ve heard good things about <a href="http://www.amazon.com/Art-Programming-Statistical-Software-Design/dp/1593273843">The Art of R Programming</a>.</li>
</ul>
</div>
Introduction to Conditional Random Fields2012-01-03T00:00:00+01:002012-01-03T00:00:00+01:00Edwin Chentag:blog.echen.me,2012-01-03:/2012/01/03/introduction-to-conditional-random-fields/<p>Imagine you have a sequence of snapshots from a day in Justin Bieber’s life, and you want to label each image with the activity it represents (eating, sleeping, driving, etc.). How can you do this?</p>
<p>One way is to ignore the sequential nature of the snapshots, and build a …</p><p>Imagine you have a sequence of snapshots from a day in Justin Bieber’s life, and you want to label each image with the activity it represents (eating, sleeping, driving, etc.). How can you do this?</p>
<p>One way is to ignore the sequential nature of the snapshots, and build a <em>per-image</em> classifier. For example, given a month’s worth of labeled snapshots, you might learn that dark images taken at 6am tend to be about sleeping, images with lots of bright colors tend to be about dancing, images of cars are about driving, and so on.</p>
<p>By ignoring this sequential aspect, however, you lose a lot of information. For example, what happens if you see a close-up picture of a mouth – is it about singing or eating? If you know that the <em>previous</em> image is a picture of Justin Bieber eating or cooking, then it’s more likely this picture is about eating; if, however, the previous image contains Justin Bieber singing or dancing, then this one probably shows him singing as well.</p>
<p>Thus, to increase the accuracy of our labeler, we should incorporate the labels of nearby photos, and this is precisely what a <strong>conditional random field</strong> does.</p>
<h1>Part-of-Speech Tagging</h1>
<p>Let’s go into some more detail, using the more common example of part-of-speech tagging.</p>
<p>In POS tagging, the goal is to label a sentence (a sequence of words or tokens) with tags like ADJECTIVE, NOUN, PREPOSITION, VERB, ADVERB, ARTICLE.</p>
<p>For example, given the sentence “Bob drank coffee at Starbucks”, the labeling might be “Bob (NOUN) drank (VERB) coffee (NOUN) at (PREPOSITION) Starbucks (NOUN)”.</p>
<p>So let’s build a conditional random field to label sentences with their parts of speech. Just like any classifier, we’ll first need to decide on a set of feature functions <span class="math">\(f_i\)</span>.</p>
<h2>Feature Functions in a CRF</h2>
<p>In a CRF, each <strong>feature function</strong> is a function that takes in as input:</p>
<ul>
<li>a sentence s</li>
<li>the position i of a word in the sentence</li>
<li>the label <span class="math">\(l_i\)</span> of the current word</li>
<li>the label <span class="math">\(l_{i-1}\)</span> of the previous word</li>
</ul>
<p>and outputs a real-valued number (though the numbers are often just either 0 or 1).</p>
<p>(Note: by restricting our features to depend on only the current and previous labels, rather than arbitrary labels throughout the sentence, I’m actually building the special case of a <strong>linear-chain CRF</strong>. For simplicity, I’m going to ignore general CRFs in this post.)</p>
<p>For example, one possible feature function could measure how much we suspect that the current word should be labeled as an adjective given that the previous word is “very”.</p>
<h2>Features to Probabilities</h2>
<p>Next, assign each feature function <span class="math">\(f_j\)</span> a weight <span class="math">\(\lambda_j\)</span> (I’ll talk below about how to learn these weights from the data). Given a sentence s, we can now score a labeling l of s by adding up the weighted features over all words in the sentence:</p>
<p><span class="math">\(score(l | s) = \sum_{j = 1}^m \sum_{i = 1}^n \lambda_j f_j(s, i, l_i, l_{i-1})\)</span></p>
<p>(The first sum runs over each feature function <span class="math">\(j\)</span>, and the inner sum runs over each position <span class="math">\(i\)</span> of the sentence.)</p>
<p>Finally, we can transform these scores into probabilities <span class="math">\(p(l | s)\)</span> between 0 and 1 by exponentiating and normalizing:</p>
<p><span class="math">\(p(l | s) = \frac{exp[score(l|s)]}{\sum_{l’} exp[score(l’|s)]} = \frac{exp[\sum_{j = 1}^m \sum_{i = 1}^n \lambda_j f_j(s, i, l_i, l_{i-1})]}{\sum_{l’} exp[\sum_{j = 1}^m \sum_{i = 1}^n \lambda_j f_j(s, i, l’_i, l’_{i-1})]}\)</span></p>
<h2>Example Feature Functions</h2>
<p>So what do these feature functions look like? Examples of POS tagging features could include:</p>
<ul>
<li>
<p><span class="math">\(f_1(s, i, l_i, l_{i-1}) = 1\)</span> if <span class="math">\(l_i =\)</span> ADVERB and the ith word ends in “-ly”; 0 otherwise.
** If the weight <span class="math">\(\lambda_1\)</span> associated with this feature is large and positive, then this feature is essentially saying that we prefer labelings where words ending in -ly get labeled as ADVERB.</p>
</li>
<li>
<p><span class="math">\(f_2(s, i, l_i, l_{i-1}) = 1\)</span> if <span class="math">\(i = 1\)</span>, <span class="math">\(l_i =\)</span> VERB, and the sentence ends in a question mark; 0 otherwise.
** Again, if the weight <span class="math">\(\lambda_2\)</span> associated with this feature is large and positive, then labelings that assign VERB to the first word in a question (e.g., “Is this a sentence beginning with a verb?”) are preferred.</p>
</li>
<li>
<p><span class="math">\(f_3(s, i, l_i, l_{i-1}) = 1\)</span> if <span class="math">\(l_{i-1} =\)</span> ADJECTIVE and <span class="math">\(l_i =\)</span> NOUN; 0 otherwise.
** Again, a positive weight for this feature means that adjectives tend to be followed by nouns.</p>
</li>
<li>
<p><span class="math">\(f_4(s, i, l_i, l_{i-1}) = 1\)</span> if <span class="math">\(l_{i-1} =\)</span> PREPOSITION and <span class="math">\(l_i =\)</span> PREPOSITION.
** A negative weight <span class="math">\(\lambda_4\)</span> for this function would mean that prepositions don’t tend to follow prepositions, so we should avoid labelings where this happens.</p>
</li>
</ul>
<p>And that’s it! To sum up: to build a conditional random field, you just define a bunch of feature functions (which can depend on the entire sentence, a current position, and nearby labels), assign them weights, and add them all together, transforming at the end to a probability if necessary.</p>
<p>Now let’s step back and compare CRFs to some other common machine learning techniques.</p>
<h1>Smells like Logistic Regression…</h1>
<p>The form of the CRF probabilities <span class="math">\(p(l | s) = \frac{exp[\sum_{j = 1}^m \sum_{i = 1}^n f_j(s, i, l_i, l_{i-1})]}{\sum_{l’} exp[\sum_{j = 1}^m \sum_{i = 1}^n f_j(s, i, l’_i, l’_{i-1})]}\)</span> might look <a href="http://en.wikipedia.org/wiki/Logistic_regression">familiar</a>.</p>
<p>That’s because CRFs are indeed basically the sequential version of logistic regression: whereas logistic regression is a log-linear model for classification, CRFs are a log-linear model for sequential labels.</p>
<h1>Looks like HMMs…</h1>
<p>Recall that Hidden Markov Models are another model for part-of-speech tagging (and sequential labeling in general). Whereas CRFs throw any bunch of functions together to get a label score, HMMs take a generative approach to labeling, defining</p>
<p><span class="math">\(p(l,s) = p(l_1) \prod_i p(l_i | l_{i-1}) p(w_i | l_i)\)</span></p>
<p>where</p>
<ul>
<li><span class="math">\(p(l_i | l_{i-1})\)</span> are transition probabilities (e.g., the probability that a preposition is followed by a noun);</li>
<li><span class="math">\(p(w_i | l_i)\)</span> are emission probabilities (e.g., the probability that a noun emits the word “dad”).</li>
</ul>
<p>So how do HMMs compare to CRFs? CRFs are more powerful – they can model everything HMMs can and more. One way of seeing this is as follows.</p>
<p>Note that the log of the HMM probability is <span class="math">\(\log p(l,s) = \log p(l_0) + \sum_i \log p(l_i | l_{i-1}) + \sum_i \log p(w_i | l_i)\)</span>. This has exactly the log-linear form of a CRF if we consider these log-probabilities to be the weights associated to binary transition and emission indicator features.</p>
<p>That is, we can build a CRF equivalent to any HMM by…</p>
<ul>
<li>For each HMM transition probability <span class="math">\(p(l_i = y | l_{i-1} = x)\)</span>, define a set of CRF transition features of the form <span class="math">\(f_{x,y}(s, i, l_i, l_{i-1}) = 1\)</span> if <span class="math">\(l_i = y\)</span> and <span class="math">\(l_{i-1} = x\)</span>. Give each feature a weight of <span class="math">\(w_{x,y} = \log p(l_i = y | l_{i-1} = x)\)</span>.</li>
<li>Similarly, for each HMM emission probability <span class="math">\(p(w_i = z | l_{i} = x)\)</span>, define a set of CRF emission features of the form <span class="math">\(g_{x,y}(s, i, l_i, l_{i-1}) = 1\)</span> if <span class="math">\(w_i = z\)</span> and <span class="math">\(l_i = x\)</span>. Give each feature a weight of <span class="math">\(w_{x,z} = \log p(w_i = z | l_i = x)\)</span>.</li>
</ul>
<p>Thus, the score <span class="math">\(p(l|s)\)</span> computed by a CRF using these feature functions is precisely proportional to the score computed by the associated HMM, and so every HMM is equivalent to some CRF.</p>
<p>However, CRFs can model a much richer set of label distributions as well, for two main reasons:</p>
<ul>
<li><strong>CRFs can define a much larger set of features.</strong> Whereas HMMs are necessarily local in nature (because they’re constrained to binary transition and emission feature functions, which force each word to depend only on the current label and each label to depend only on the previous label), CRFs can use more global features. For example, one of the features in our POS tagger above increased the probability of labelings that tagged the first word of a sentence as a VERB if the end of the sentence contained a question mark.</li>
<li><strong>CRFs can have arbitrary weights.</strong> Whereas the probabilities of an HMM must satisfy certain constraints (e.g., <span class="math">\(0 <= p(w_i | l_i) <= 1, \sum_w p(w_i = w | l_1) = 1)\)</span>, the weights of a CRF are unrestricted (e.g., <span class="math">\(\log p(w_i | l_i)\)</span> can be anything it wants).</li>
</ul>
<h1>Learning Weights</h1>
<p>Let’s go back to the question of how to learn the feature weights in a CRF. One way, unsurprisingly, is to use <strong>gradient descent</strong>.</p>
<p>Assume we have a bunch of training examples (sentences and associated part-of-speech labels). Randomly initialize the weights of our CRF model. To shift these randomly initialized weights to the correct ones, for each training example…</p>
<ul>
<li>Go through each feature function <span class="math">\(f_i\)</span>, and calculate the gradient of the log probability of the training example with respect to <span class="math">\(\lambda_i\)</span>: <span class="math">\(\frac{\partial}{\partial w_j} \log p(l | s) = \sum_{j = 1}^m f_i(s, j, l_j, l_{j-1}) - \sum_{l’} p(l’ | s) \sum_{j = 1}^m f_i(s, j, l’_j, l’_{j-1})\)</span></li>
<li>Note that the first term in the gradient is the contribution of feature <span class="math">\(f_i\)</span> under the true label, and the second term in the gradient is the expected contribution of feature <span class="math">\(f_i\)</span> under the current model. This is exactly the form you’d expect gradient ascent to take.</li>
<li>Move <span class="math">\(\lambda_i\)</span> in the direction of the gradient: <span class="math">\(\lambda_i = \lambda_i + \alpha [\sum_{j = 1}^m f_i(s, j, l_j, l_{j-1}) - \sum_{l’} p(l’ | s) \sum_{j = 1}^m f_i(s, j, l’_j, l’_{j-1})]\)</span> where <span class="math">\(\alpha\)</span> is some learning rate.</li>
<li>Repeat the previous steps until some stopping condition is reached (e.g., the updates fall below some threshold).</li>
</ul>
<p>In other words, every step takes the difference between what we want the model to learn and the model’s current state, and moves <span class="math">\(\lambda_i\)</span> in the direction of this difference.</p>
<h1>Finding the Optimal Labeling</h1>
<p>Suppose we’ve trained our CRF model, and now a new sentence comes in. How do we do label it?</p>
<p>The naive way is to calculate <span class="math">\(p(l | s)\)</span> for every possible labeling l, and then choose the label that maximizes this probability. However, since there are <span class="math">\(k^m\)</span> possible labels for a tag set of size k and a sentence of length m, this approach would have to check an exponential number of labels.</p>
<p>A better way is to realize that (linear-chain) CRFs satisfy an <a href="http://en.wikipedia.org/wiki/Optimal_substructure">optimal substructure</a> property that allows us to use a (polynomial-time) dynamic programming algorithm to find the optimal label, similar to the <a href="http://en.wikipedia.org/wiki/Viterbi_algorithm">Viterbi algorithm</a> for HMMs.</p>
<h1>A More Interesting Application</h1>
<p>Okay, so part-of-speech tagging is kind of boring, and there are plenty of existing POS taggers out there. When might you use a CRF in real life?</p>
<p>Suppose you want to mine Twitter for the types of presents people received for Christmas:</p>
<blockquote class="twitter-tweet"><p>What people on Twitter wanted for Christmas, and what they got: <a href="http://t.co/EGeKTBgF" title="http://twitter.com/edchedch/status/153683967315419136/photo/1">twitter.com/edchedch/statu…</a></p>— Edwin Chen (@echen) <a href="https://twitter.com/edchedch/status/153683967315419136" data-datetime="2012-01-02T03:48:10+00:00">January 2, 2012</a></blockquote>
<script src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>How can you figure out which words refer to gifts?</p>
<p>To gather data for the graphs above, I simply looked for phrases of the form “I want XXX for Christmas” and “I got XXX for Christmas”. However, a more sophisticated CRF variant could use a GIFT part-of-speech-like tag (even adding other tags like GIFT-GIVER and GIFT-RECEIVER, to get even more information on who got what from whom) and treat this like a POS tagging problem. Features could be based around things like “this word is a GIFT if the previous word was a GIFT-RECEIVER and the word before that was ‘gave’” or “this word is a GIFT if the next two words are ‘for Christmas’”.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Winning the Netflix Prize: A Summary2011-10-24T04:15:00+02:002011-10-24T04:15:00+02:00Edwin Chentag:blog.echen.me,2011-10-24:/2011/10/24/winning-the-netflix-prize-a-summary/
<div class="entry-content"><p>How was the <a href="http://en.wikipedia.org/wiki/Netflix_Prize">Netflix Prize</a> won? I went through a lot of the Netflix Prize papers a couple years ago, so I’ll try to give an overview of the techniques that went into the winning solution here.</p>
<h1>Normalization of Global Effects</h1>
<p>Suppose Alice rates Inception 4 stars. We can …</p></div>
<div class="entry-content"><p>How was the <a href="http://en.wikipedia.org/wiki/Netflix_Prize">Netflix Prize</a> won? I went through a lot of the Netflix Prize papers a couple years ago, so I’ll try to give an overview of the techniques that went into the winning solution here.</p>
<h1>Normalization of Global Effects</h1>
<p>Suppose Alice rates Inception 4 stars. We can think of this rating as composed of several parts:</p>
<ul>
<li>A <strong>baseline rating</strong> (e.g., maybe the mean over all user-movie ratings is 3.1 stars).</li>
<li>An <strong>Alice-specific effect</strong> (e.g., maybe Alice tends to rate movies lower than the average user, so her ratings are -0.5 stars lower than we normally expect).</li>
<li>An <strong>Inception-specific effect</strong> (e.g., Inception is a pretty awesome movie, so its ratings are 0.7 stars higher than we normally expect).</li>
<li>A less predictable effect based on the <strong>specific interaction</strong> between Alice and Inception that accounts for the remainder of the stars (e.g., Alice really liked Inception because of its particular combination of Leonardo DiCaprio and neuroscience, so this rating gets an additional 0.7 stars).</li>
</ul>
<p>In other words, we’ve decomposed the 4-star rating into:
4 = [3.1 (the baseline rating) - 0.5 (the Alice effect) + 0.7 (the Inception effect)] + 0.7 (the specific interaction)</p>
<p>So instead of having our models predict the 4-star rating itself, we could first try to remove the effect of the baseline predictors (the first three components) and have them predict the specific 0.7 stars. (I guess you can also think of this as a simple kind of boosting.)</p>
<p>More generally, additional baseline predictors include:</p>
<ul>
<li>A factor that allows Alice’s rating to (linearly) depend on the (square root of the) <strong>number of days since her first rating</strong>. (For example, have you ever noticed that you become a harsher critic over time?)</li>
<li>A factor that allows Alice’s rating to depend on the <strong>number of days since the movie’s first rating by anyone</strong>. (If you’re one of the first people to watch it, maybe it’s because you’re a huge fan and really excited to see it on DVD, so you’ll tend to rate it higher.)</li>
<li>A factor that allows Alice’s rating to depend on the <strong>number of people who have rated Inception</strong>. (Maybe Alice is a hipster who hates being part of the crowd.)</li>
<li>A factor that allows Alice’s rating to <strong>depend on the movie’s overall rating</strong>.</li>
<li>(Plus a bunch of others.)</li>
</ul>
<p>And, in fact, modeling these biases turned out to be fairly important: in their paper describing their final solution to the Netflix Prize, Bell and Koren write that</p>
<blockquote><p>Of the numerous new algorithmic contributions, I would like to highlight one – those humble baseline predictors (or biases), which capture main effects in the data. While the literature mostly concentrates on the more sophisticated algorithmic aspects, we have learned that an accurate treatment of main effects is probably at least as signficant as coming up with modeling breakthroughs.</p></blockquote>
<p>(For a perhaps more concrete example of why removing these biases is useful, suppose you know that Bob likes the same kinds of movies that Alice does. To predict Bob’s rating of Inception, instead of simply predicting the same 4 stars that Alice rated, if we know that Bob tends to rate movies 0.3 stars higher than average, then we could first remove Alice’s bias and then add in Bob’s: 4 + 0.5 + 0.3 = 4.8.)</p>
<h1>Neighborhood Models</h1>
<p>Let’s now look at some slightly more sophisticated models. As alluded to in the section above, one of the standard approaches to collaborative filtering is to use neighborhood models.</p>
<p>Briefly, a neighborhood model works as follows. To predict Alice’s rating of Titanic, you could do two things:</p>
<ul>
<li><strong>Item-item approach</strong>: find a set of items similar to Titanic that Alice has also rated, and take the (weighted) mean of Alice’s ratings on them.</li>
<li><strong>User-user approach</strong>: find a set of users similar to Alice who rated Titanic, and again take the mean of their ratings of Titanic.</li>
</ul>
<p>(See also my post on <a href="http://blog.echen.me/2011/02/15/an-overview-of-item-to-item-collaborative-filtering-with-amazons-recommendation-system/">item-to-item collaborative filtering on Amazon</a>.)</p>
<p>The main questions, then, are (let’s stick to the item-item approach for simplicity):</p>
<ul>
<li>How do we find the set of similar items?</li>
<li>How do we weight these items when taking their mean?</li>
</ul>
<p>The standard approach is to take some similarity metric (e.g., correlation or a Jaccard index) to define similarities between pairs of movies, take the K most similar movies under this metric (where K is perhaps chosen via cross-validation), and then use the same similarity metric when computing the weighted mean.</p>
<p>This has a couple problems:</p>
<ul>
<li><strong>Neighbors aren’t independent</strong>, so using a standard similarity metric to define a weighted mean overcounts information. For example, suppose you ask five friends where you should eat tonight. Three of them went to Mexico last week and are sick of burritos, so they strongly recommend against a taqueria. Thus, your friends’ recommendations have a stronger bias than what you’d get if you asked five friends who didn’t know each other at all. (Compare with the situation where all three Lord of the Rings Movies are neighbors of Harry Potter.)</li>
<li>Different movies should perhaps be using <strong>different numbers of neighbors</strong>. Some movies may be predicted well by only one neighbor (e.g., Harry Potter 2 could be predicted well by Harry Potter 1 alone), some movies may require more, and some movies may have no good neighbors (so you should ignore your neighborhood algorithms entirely and let your other ratings models stand on their own).</li>
</ul>
<p>So another approach is the following:</p>
<ul>
<li>You can still use a similarity metric like correlation or cosine similarity to choose the set of similar items.</li>
<li>But instead of using the similarity metric to define the interpolation weights in the mean calculations, you essentially perform a (sparse) <strong>linear regression to find the weights</strong> that minimize the squared error between an item’s rating and a linear combination of the ratings of its neighbors. Note that these weights are no longer constrained, so that if all neighbors are weak, then their weights will be close to zero and the neighborhood model will have a low effect.</li>
</ul>
<p>(A slightly more complicated user-user approach, similar to this item-item neighborhood approach, is also useful.)</p>
<h1>Implicit Data</h1>
<p>Adding on to the neighborhood approach, we can also let <strong>implicit data influence our predictions</strong>. The mere fact that a user rated lots of science fiction movies but no westerns, suggests that the user likes science fiction better than cowboys. So using a similar framework as in the neighborhood ratings model, we can learn for Inception a set of <strong>offset weights</strong> associated to Inception’s movie neighbors.</p>
<p>Whenever we want to predict how Bob rates Inception, we look at whether Bob rated each of Inception’s neighbors. If he did, we add in the corresponding offset; if not, then we add nothing (and, thus, Bob’s rating is implicitly penalized by the missing weight).</p>
<h1>Matrix Factorization</h1>
<p>Complementing the neighborhood approach to collaborative filtering is the matrix factorization approach. Whereas the neighborhood approach takes a very local approach to ratings (if you liked Harry Potter 1, then you’ll like Harry Potter 2!), the factorization approach takes a more global view (we know that you like fantasy movies and that Harry Potter has a strong fantasy element, so we think that you’ll like Harry Potter) that <strong>decomposes users and movies into a set of latent factors</strong> (which we can think of as categories like “fantasy” or “violence”).</p>
<p>In fact, matrix factorization methods were probably the most important class of techniques for winning the Netflix Prize. In their 2008 Progress Prize paper, Bell and Koren write</p>
<blockquote><p>It seems that models based on matrix-factorization were found to be most accurate (and thus popular), as evident by recent publications and discussions on the Netflix Prize forum. We definitely agree to that, and would like to add that those matrix-factorization models also offer the important flexibility needed for modeling temporal effects and the binary view. Nonetheless, neighborhood models, which have been dominating most of the collaborative filtering literature, are still expected to be popular due to their practical characteristics - being able to handle new users/ratings without re-training and offering direct explanations to the recommendations.</p></blockquote>
<p>The typical way to perform matrix factorizations is to perform a <strong>singular value decomposition</strong> on the (sparse) ratings matrix (using stochastic gradient descent and regularizing the weights of the factors, possibly constraining the weights to be positive to get a type of non-negative matrix factorization). (Note that this “SVD” is a little different from the standard SVD learned in linear algebra, since not every user has rated every movie and so the ratings matrix contains many missing elements that we don’t want to simply treat as 0.)</p>
<p>Some SVD-inspired methods used in the Netflix Prize include:</p>
<ul>
<li><strong>Standard SVD</strong>: Once you’ve represented users and movies as factor vectors, you can dot product Alice’s vector with Inception’s vector to get Alice’s predicted rating of Inception.</li>
<li><strong>Asymmetric SVD</strong>: Instead of users having their own notion of factor vectors, we can represent users as a bag of items they have rated (or provided implicit feedback for). So Alice is now represented as a (possibly weighted) sum of the factor vectors of the items she has rated, and to get her predicted rating of Titanic, we can dot product this representation with the factor vector of Titanic. From a practical perspective, this model has an added benefit in that no user parameterizations are needed, so we can use this approach to generate recommendations as soon as a user provides some feedback (which could just be views or clicks on an item, and not necessarily ratings), without needing to retrain the model to factorize the user.</li>
<li><strong>SVD++</strong>: Incorporate both the standard SVD and the asymmetric SVD model by representing users both by their own factor representation and as a bag of item vectors.</li>
</ul>
<h1>Regression</h1>
<p>Some regression models were also used in the predictions. The models are fairly standard, I think, so I won’t spend too long here. Basically, just as with the neighborhood models, we can take a user-centric approach and a movie-centric approach to regression:</p>
<ul>
<li><strong>User-centric approach</strong>: We learn a regression model for each user, using all the movies that the user rated as the dataset. The response is the movie’s rating, and the predictor variables are attributes associated to that movie (which can be derived from, say, PCA, MDS, or an SVD).</li>
<li><strong>Movie-centric approach</strong>: Similarly, we can learn a regression model for each movie, using all the users that rated the movie as the dataset.</li>
</ul>
<h1>Restricted Boltzmann Machines</h1>
<p>Restricted Boltzmann Machines provide another kind of <strong>latent factor approach</strong> that can be used. See <a href="http://www.machinelearning.org/proceedings/icml2007/papers/407.pdf">this paper</a> for a description of how to apply them to the Netflix Prize. (In case the paper’s a little difficult to read, I wrote an <a href="http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines">introduction to RBMs</a> a little while ago.)</p>
<h1>Temporal Effects</h1>
<p>Many of the models incorporate temporal effects. For example, when describing the baseline predictors above, we used a few temporal predictors that allowed a user’s rating to (linearly) depend on the time since the first rating he ever made and on the time since a movie’s first rating. We can also get more fine-grained temporal effects by, say, binning items into a couple months’ worth of ratings at a time, and allowing movie biases to change within each bin. (For example, maybe in May 2006, Time Magazine nominated Titanic as the best movie ever made, which caused a spurt in glowing ratings around that time.)</p>
<p>In the matrix factorization approach, user factors were also allowed to be time-dependent (e.g., maybe Bob comes to like comedy movies more and more over time). We can also give more weight to recent user actions.</p>
<h1>Regularization</h1>
<p>Regularization was also applied throughout pretty much all the models learned, to <strong>prevent overfitting</strong> on the dataset. Ridge regression was heavily used in the factorization models to penalize large weights, and lasso regression (though less effective) was useful as well. Many other parameters (e.g., the baseline predictors, similarity weights and interpolation weights in the neighborhood models) were also estimated using fairly standard shrinkage techniques.</p>
<h1>Ensemble Methods</h1>
<p>Finally, let’s talk about how all of these different algorithms were combined to provide a single rating that <strong>exploits the strengths of each model</strong>. (Note that, as mentioned above, many of these models were not trained on the raw ratings data directly, but rather on the residuals of other models.)</p>
<p>In the paper detailing their final solution, the winners describe using <strong>gradient boosted decision trees to combine over 500 models</strong>; previous solutions used instead a <strong>linear regression</strong> to combine the predictors.</p>
<p>Briefly, gradient boosted decision trees work by sequentially fitting a series of decision trees to the data; each tree is asked to predict the error made by the previous trees, and is often trained on slightly perturbed versions of the data. (For a longer description of a similar technique, see <a href="http://blog.echen.me/2011/03/14/laymans-introduction-to-random-forests/">my introduction to random forests</a>.)</p>
<p>Since GBDTs have a built-in ability to apply different methods to different slices of the data, we can add in some predictors that help the trees make useful clusterings:</p>
<ul>
<li>Number of movies each user rated</li>
<li>Number of users that rated each movie</li>
<li>Factor vectors of users and movies</li>
<li>Hidden units of a restricted Boltzmann Machine</li>
</ul>
<p>(For example, one thing that Bell and Koren found (when using an earlier ensemble method) was that RBMs are more useful when the movie or the user has a low number of ratings, and that matrix factorization methods are more useful when the movie or user has a high number of ratings.)</p>
<p>Here’s a graph of the effect of ensemble size from early on in the competition (in 2007), and the authors’ take on it:</p>
<p><a href="http://www2.research.att.com/~volinsky/netflix/newensemble.gif"><img src="http://www2.research.att.com/~volinsky/netflix/newensemble.gif" alt="Ensemble Size vs. RMSE" /></a></p>
<blockquote><p>However, we would like to stress that it is not necessary to have such a large number of models to do well. The plot below shows RMSE as a function of the number of methods used. One can achieve our winning score (RMSE=0.8712) with less than 50 methods, using the best 3 methods can yield RMSE < 0.8800, which would land in the top 10. Even just using our single best method puts us on the leaderboard with an RMSE of 0.8890. The lesson here is that having lots of models is useful for the incremental results needed to win competitions, but practically, excellent systems can be built with just a few well-selected models.</p></blockquote>
</div>
Stuff Harvard People Like2011-09-29T04:15:00+02:002011-09-29T04:15:00+02:00Edwin Chentag:blog.echen.me,2011-09-29:/2011/09/29/stuff-harvard-people-like/
<div class="entry-content"><p>What types of students go to which schools? There are, of course, the classic stereotypes:</p>
<ul>
<li><strong>MIT</strong> has the hacker engineers.</li>
<li><strong>Stanford</strong> has the laid-back, social folks.</li>
<li><strong>Harvard</strong> has the prestigious leaders of the world.</li>
<li><strong>Berkeley</strong> has the activist hippies.</li>
<li><strong>Caltech</strong> has the hardcore science nerds.</li>
</ul>
<p>But how well do these …</p></div>
<div class="entry-content"><p>What types of students go to which schools? There are, of course, the classic stereotypes:</p>
<ul>
<li><strong>MIT</strong> has the hacker engineers.</li>
<li><strong>Stanford</strong> has the laid-back, social folks.</li>
<li><strong>Harvard</strong> has the prestigious leaders of the world.</li>
<li><strong>Berkeley</strong> has the activist hippies.</li>
<li><strong>Caltech</strong> has the hardcore science nerds.</li>
</ul>
<p>But how well do these perceptions match reality? What are students at Stanford, Harvard, MIT, Caltech, and Berkeley <em>really</em> interested in? Following the path of my previous data-driven post on <a href="http://blog.echen.me/2011/04/18/twifferences-between-californians-and-new-yorkers/">differences between Silicon Valley and NYC</a>, I scraped the Quora profiles of a couple hundred followers of each school to find out.</p>
<h1>Topics</h1>
<p>So let’s look at what kinds of topics followers of each school are interested in*. (Skip past the lists for a discussion.)</p>
<h2>MIT</h2>
<p>Topics are followed by p(school = MIT|topic).</p>
<ul>
<li><strong>MIT Media Lab</strong> 0.893</li>
<li><strong>Ksplice</strong> 0.69</li>
<li><strong>Lisp (programming language)</strong> 0.677</li>
<li><strong>Nokia</strong> 0.659</li>
<li><strong>Public Speaking</strong> 0.65</li>
<li><strong>Data Storage</strong> 0.65</li>
<li><strong>Google Voice</strong> 0.609</li>
<li><strong>Hacking</strong> 0.602</li>
<li><strong>Startups in Europe</strong> 0.597</li>
<li><strong>Startup Names</strong> 0.572</li>
<li><strong>Mechanical Engineering</strong> 0.563</li>
<li><strong>Engineering</strong> 0.563</li>
<li><strong>Distributed Databases</strong> 0.544</li>
<li><strong>StackOverflow</strong> 0.536</li>
<li><strong>Boston</strong> 0.513</li>
<li><strong>Learning</strong> 0.507</li>
<li><strong>Open Source</strong> 0.498</li>
<li><strong>Cambridge</strong> 0.496</li>
<li><strong>Public Relations</strong> 0.493</li>
<li><strong>Visualization</strong> 0.492</li>
<li><strong>Semantic Web</strong> 0.486</li>
<li><strong>Andreessen-Horowitz</strong> 0.483</li>
<li><strong>Nature</strong> 0.475</li>
<li><strong>Cryptography</strong> 0.474</li>
<li><strong>Startups in Boston</strong> 0.452</li>
<li><strong>Adobe Photoshop</strong> 0.451</li>
<li><strong>Computer Security</strong> 0.447</li>
<li><strong>Sachin Tendulkar</strong> 0.443</li>
<li><strong>Hacker News</strong> 0.442</li>
<li><strong>Games</strong> 0.429</li>
<li><strong>Android Applications</strong> 0.428</li>
<li><strong>Best Engineers and Programmers</strong> 0.427</li>
<li><strong>College Admissions & Getting Into College</strong> 0.422</li>
<li><strong>Co-Founders</strong> 0.419</li>
<li><strong>Big Data</strong> 0.41</li>
<li><strong>System Administration</strong> 0.4</li>
<li><strong>Biotechnology</strong> 0.398</li>
<li><strong>Higher Education</strong> 0.394</li>
<li><strong>NoSQL</strong> 0.387</li>
<li><strong>User Experience</strong> 0.386</li>
<li><strong>Career Advice</strong> 0.377</li>
<li><strong>Artificial Intelligence</strong> 0.375</li>
<li><strong>Scalability</strong> 0.37</li>
<li><strong>Taylor Swift</strong> 0.368</li>
<li><strong>Google Search</strong> 0.368</li>
<li><strong>Functional Programming</strong> 0.365</li>
<li><strong>Bing</strong> 0.363</li>
<li><strong>Bioinformatics</strong> 0.361</li>
<li><strong>How I Met Your Mother (TV series)</strong> 0.361</li>
<li><strong>Operating Systems</strong> 0.356</li>
<li><strong>Compilers</strong> 0.355</li>
<li><strong>Google Chrome</strong> 0.354</li>
<li><strong>Management & Organizational Leadership</strong> 0.35</li>
<li><strong>Literary Fiction</strong> 0.35</li>
<li><strong>Intelligence</strong> 0.348</li>
<li><strong>Fight Club (1999 movie)</strong> 0.344</li>
<li><strong>Hip Hop Music</strong> 0.34</li>
<li><strong>UX Design</strong> 0.337</li>
<li><strong>Web Application Frameworks</strong> 0.336</li>
<li><strong>Startups in New York City</strong> 0.333</li>
<li><strong>Book Recommendations</strong> 0.33</li>
<li><strong>Engineering Recruiting</strong> 0.33</li>
<li><strong>Search Engines</strong> 0.329</li>
<li><strong>Social Search</strong> 0.329</li>
<li><strong>Data Science</strong> 0.328</li>
<li><strong>History</strong> 0.328</li>
<li><strong>Interaction Design</strong> 0.326</li>
<li><strong>Classification (machine learning)</strong> 0.322</li>
<li><strong>Startup Incubators and Seed Programs</strong> 0.321</li>
<li><strong>Graphic Design</strong> 0.321</li>
<li><strong>Product Design (software)</strong> 0.319</li>
<li><strong>The College Experience</strong> 0.319</li>
<li><strong>Writing</strong> 0.319</li>
<li><strong>MapReduce</strong> 0.318</li>
<li><strong>Database Systems</strong> 0.315</li>
<li><strong>User Interfaces</strong> 0.314</li>
<li><strong>Literature</strong> 0.314</li>
<li><strong>C (programming language)</strong> 0.314</li>
<li><strong>Television</strong> 0.314</li>
<li><strong>Reading</strong> 0.313</li>
<li><strong>Usability</strong> 0.312</li>
<li><strong>Books</strong> 0.312</li>
<li><strong>Computers</strong> 0.311</li>
<li><strong>Stealth Startups</strong> 0.311</li>
<li><strong>Daft Punk</strong> 0.31</li>
<li><strong>Healthy Eating</strong> 0.309</li>
<li><strong>Innovation</strong> 0.309</li>
<li><strong>Skiing</strong> 0.305</li>
<li><strong>JavaScript</strong> 0.304</li>
<li><strong>Rock Music</strong> 0.304</li>
<li><strong>Mozilla Firefox</strong> 0.304</li>
<li><strong>Self-Improvement</strong> 0.303</li>
<li><strong>McKinsey & Company</strong> 0.302</li>
<li><strong>AngelList</strong> 0.301</li>
<li><strong>Data Visualization</strong> 0.301</li>
<li><strong>Cassandra (database)</strong> 0.301</li>
</ul>
<h2>Stanford</h2>
<p>Topics are followed by p(school = Stanford|topic).</p>
<ul>
<li><strong>Stanford Computer Science</strong> 0.951</li>
<li><strong>Stanford Graduate School of Business</strong> 0.939</li>
<li><strong>Stanford</strong> 0.896</li>
<li><strong>Stanford Football</strong> 0.896</li>
<li><strong>Stanford Cardinal</strong> 0.896</li>
<li><strong>Social Dance</strong> 0.847</li>
<li><strong>Stanford University Courses</strong> 0.847</li>
<li><strong>Romance</strong> 0.769</li>
<li><strong>Instagram</strong> 0.745</li>
<li><strong>College Football</strong> 0.665</li>
<li><strong>Mobile Location Applications</strong> 0.634</li>
<li><strong>Online Communities</strong> 0.621</li>
<li><strong>Interpersonal Relationships</strong> 0.585</li>
<li><strong>Food & Restaurants in Palo Alto</strong> 0.572</li>
<li><strong>Your 20s</strong> 0.566</li>
<li><strong>Men’s Fashion</strong> 0.548</li>
<li><strong>Flipboard</strong> 0.537</li>
<li><strong>Inception (2010 movie)</strong> 0.535</li>
<li><strong>Tumblr</strong> 0.531</li>
<li><strong>People Skills</strong> 0.522</li>
<li><strong>Exercise</strong> 0.52</li>
<li><strong>Joel Spolsky</strong> 0.516</li>
<li><strong>Valuations</strong> 0.515</li>
<li><strong>The Social Network (2010 movie)</strong> 0.513</li>
<li><strong>LeBron James</strong> 0.506</li>
<li><strong>Northern California</strong> 0.506</li>
<li><strong>Evernote</strong> 0.5</li>
<li><strong>Quora Community</strong> 0.5</li>
<li><strong>Blogging</strong> 0.49</li>
<li><strong>Downtown Palo Alto</strong> 0.487</li>
<li><strong>The College Experience</strong> 0.485</li>
<li><strong>Consumer Internet</strong> 0.477</li>
<li><strong>Restaurants in San Francisco</strong> 0.477</li>
<li><strong>Chad Hurley</strong> 0.47</li>
<li><strong>Meditation</strong> 0.468</li>
<li><strong>Yishan Wong</strong> 0.466</li>
<li><strong>Arrested Development (TV series)</strong> 0.463</li>
<li><strong>fbFund</strong> 0.457</li>
<li><strong>Best Engineers at X Company</strong> 0.451</li>
<li><strong>Language</strong> 0.45</li>
<li><strong>Words</strong> 0.448</li>
<li><strong>Happiness</strong> 0.447</li>
<li><strong>Path (company)</strong> 0.446</li>
<li><strong>Color Labs (startup)</strong> 0.446</li>
<li><strong>Palo Alto</strong> 0.445</li>
<li><strong>Woot.com</strong> 0.442</li>
<li><strong>Beer</strong> 0.442</li>
<li><strong>PayPal</strong> 0.441</li>
<li><strong>Women in Startups</strong> 0.438</li>
<li><strong>Techmeme</strong> 0.433</li>
<li><strong>Women in Engineering</strong> 0.428</li>
<li><strong>The Mission (San Francisco neighborhood)</strong> 0.427</li>
<li><strong>iPhone Applications</strong> 0.416</li>
<li><strong>Asana</strong> 0.413</li>
<li><strong>Monetization</strong> 0.412</li>
<li><strong>Repetitive Strain Injury (RSI)</strong> 0.4</li>
<li><strong>IDEO</strong> 0.398</li>
<li><strong>Spotify</strong> 0.397</li>
<li><strong>San Francisco Giants</strong> 0.396</li>
<li><strong>Fortune Magazine</strong> 0.389</li>
<li><strong>Love</strong> 0.387</li>
<li><strong>Human-Computer Interaction</strong> 0.382</li>
<li><strong>Hip Hop Music</strong> 0.378</li>
<li><strong>Self-Improvement</strong> 0.378</li>
<li><strong>Food in San Francisco</strong> 0.375</li>
<li><strong>Quora (company)</strong> 0.374</li>
<li><strong>Quora Infrastructure</strong> 0.373</li>
<li><strong>iPhone</strong> 0.371</li>
<li><strong>Square (company)</strong> 0.369</li>
<li><strong>Social Psychology</strong> 0.369</li>
<li><strong>Network Effects</strong> 0.366</li>
<li><strong>Chris Sacca</strong> 0.365</li>
<li><strong>Walt Mossberg</strong> 0.364</li>
<li><strong>Salesforce.com</strong> 0.362</li>
<li><strong>Sex</strong> 0.361</li>
<li><strong>Etiquette</strong> 0.361</li>
<li><strong>David Pogue</strong> 0.361</li>
<li><strong>Gowalla</strong> 0.36</li>
<li><strong>iOS Development</strong> 0.354</li>
<li><strong>Palantir Technologies</strong> 0.353</li>
<li><strong>Mobile Computing</strong> 0.347</li>
<li><strong>Sports</strong> 0.346</li>
<li><strong>Video Games</strong> 0.345</li>
<li><strong>Burning Man</strong> 0.345</li>
<li><strong>Engineering Management</strong> 0.343</li>
<li><strong>Cognitive Science</strong> 0.342</li>
<li><strong>Dating & Relationships</strong> 0.341</li>
<li><strong>Fred Wilson (venture investor)</strong> 0.337</li>
<li><strong>Taiwan</strong> 0.333</li>
<li><strong>Natural Language Processing</strong> 0.33</li>
<li><strong>Eric Schmidt</strong> 0.329</li>
<li><strong>Social Advice</strong> 0.329</li>
<li><strong>Engineering Recruiting</strong> 0.328</li>
<li><strong>Job Interviews</strong> 0.325</li>
<li><strong>Mobile Phones</strong> 0.324</li>
<li><strong>Twitter Inc. (company)</strong> 0.321</li>
<li><strong>Engineering in Silicon Valley</strong> 0.321</li>
<li><strong>San Francisco Bay Area</strong> 0.321</li>
<li><strong>Google Analytics</strong> 0.32</li>
<li><strong>Fashion</strong> 0.315</li>
<li><strong>Interaction Design</strong> 0.314</li>
<li><strong>Open Graph</strong> 0.313</li>
<li><strong>Drugs & Pharmaceuticals</strong> 0.312</li>
<li><strong>Electronic Music</strong> 0.312</li>
<li><strong>Facebook Inc. (company)</strong> 0.309</li>
<li><strong>Fitness</strong> 0.309</li>
<li><strong>YouTube</strong> 0.308</li>
<li><strong>TED Talks</strong> 0.308</li>
<li><strong>Freakonomics (2005 Book)</strong> 0.307</li>
<li><strong>Jack Dorsey</strong> 0.306</li>
<li><strong>Nutrition</strong> 0.305</li>
<li><strong>Puzzles</strong> 0.305</li>
<li><strong>Silicon Valley Mergers & Acquisitions</strong> 0.304</li>
<li><strong>Viral Growth & Analytics</strong> 0.304</li>
<li><strong>Amazon Web Services</strong> 0.304</li>
<li><strong>StumbleUpon</strong> 0.303</li>
<li><strong>Exceptional Comment Threads</strong> 0.303</li>
</ul>
<h2>Harvard</h2>
<ul>
<li><strong>Harvard Business School</strong> 0.968</li>
<li><strong>Harvard Business Review</strong> 0.922</li>
<li><strong>Harvard Square</strong> 0.912</li>
<li><strong>Harvard Law School</strong> 0.912</li>
<li><strong>Jimmy Fallon</strong> 0.899</li>
<li><strong>Boston Red Sox</strong> 0.658</li>
<li><strong>Klout</strong> 0.644</li>
<li><strong>Oprah Winfrey</strong> 0.596</li>
<li><strong>Ivanka Trump</strong> 0.587</li>
<li><strong>Dalai Lama</strong> 0.569</li>
<li><strong>Food in New York City</strong> 0.565</li>
<li><strong>U2</strong> 0.562</li>
<li><strong>TwitPic</strong> 0.534</li>
<li><strong>37signals</strong> 0.522</li>
<li><strong>David Lynch (director)</strong> 0.512</li>
<li><strong>Al Gore</strong> 0.508</li>
<li><strong>TechStars</strong> 0.49</li>
<li><strong>Baseball</strong> 0.487</li>
<li><strong>Private Equity</strong> 0.471</li>
<li><strong>Classical Music</strong> 0.46</li>
<li><strong>Startups in New York City</strong> 0.458</li>
<li><strong>HootSuite</strong> 0.449</li>
<li><strong>Kiva</strong> 0.442</li>
<li><strong>Ultimate Frisbee</strong> 0.441</li>
<li><strong>Huffington Post</strong> 0.436</li>
<li><strong>New York City</strong> 0.433</li>
<li><strong>Charlie Cheever</strong> 0.433</li>
<li><strong>The New York Times</strong> 0.431</li>
<li><strong>Technology Journalism</strong> 0.431</li>
<li><strong>McKinsey & Company</strong> 0.427</li>
<li><strong>TweetDeck</strong> 0.422</li>
<li><strong>How Does X Work?</strong> 0.417</li>
<li><strong>Ashton Kutcher</strong> 0.414</li>
<li><strong>Coldplay</strong> 0.402</li>
<li><strong>Conan O’Brien</strong> 0.397</li>
<li><strong>Fast Company</strong> 0.397</li>
<li><strong>WikiLeaks</strong> 0.394</li>
<li><strong>Michael Jackson</strong> 0.389</li>
<li><strong>Guy Kawasaki</strong> 0.389</li>
<li><strong>Journalism</strong> 0.384</li>
<li><strong>Wall Street Journal</strong> 0.384</li>
<li><strong>Cambridge</strong> 0.371</li>
<li><strong>Seattle</strong> 0.37</li>
<li><strong>Cities & Metro Areas</strong> 0.357</li>
<li><strong>Boston</strong> 0.353</li>
<li><strong>Tim Ferriss (author)</strong> 0.35</li>
<li><strong>The New Yorker</strong> 0.343</li>
<li><strong>Law</strong> 0.34</li>
<li><strong>Mashable</strong> 0.338</li>
<li><strong>Politics</strong> 0.335</li>
<li><strong>The Economist</strong> 0.334</li>
<li><strong>Barack Obama</strong> 0.333</li>
<li><strong>Skiing</strong> 0.329</li>
<li><strong>McKinsey Quarterly</strong> 0.325</li>
<li><strong>Wired (magazine)</strong> 0.316</li>
<li><strong>Bill Gates</strong> 0.31</li>
<li><strong>Mad Men (TV series)</strong> 0.308</li>
<li><strong>India</strong> 0.306</li>
<li><strong>TED Talks</strong> 0.306</li>
<li><strong>Netflix</strong> 0.304</li>
<li><strong>Wine</strong> 0.303</li>
<li><strong>Angel Investors</strong> 0.302</li>
<li><strong>Facebook Ads</strong> 0.301</li>
</ul>
<h2>UC Berkeley</h2>
<ul>
<li><strong>Berkeley</strong> 0.978</li>
<li><strong>California Golden Bears</strong> 0.91</li>
<li><strong>Internships</strong> 0.717</li>
<li><strong>Web Marketing</strong> 0.484</li>
<li><strong>Google Social Strategy</strong> 0.453</li>
<li><strong>Southwest Airlines</strong> 0.451</li>
<li><strong>WordPress</strong> 0.429</li>
<li><strong>Stock Market</strong> 0.429</li>
<li><strong>BMW (automobile)</strong> 0.428</li>
<li><strong>Web Applications</strong> 0.423</li>
<li><strong>Flickr</strong> 0.422</li>
<li><strong>Snowboarding</strong> 0.42</li>
<li><strong>Electronic Music</strong> 0.404</li>
<li><strong>MySQL</strong> 0.401</li>
<li><strong>Internet Advertising</strong> 0.399</li>
<li><strong>Search Engine Optimization (SEO)</strong> 0.398</li>
<li><strong>Yelp</strong> 0.396</li>
<li><strong>Groupon</strong> 0.393</li>
<li><strong>In-N-Out Burger</strong> 0.391</li>
<li><strong>The Matrix (1999 movie)</strong> 0.389</li>
<li><strong>Trading (finance)</strong> 0.385</li>
<li><strong>jQuery</strong> 0.381</li>
<li><strong>Hedge Funds</strong> 0.378</li>
<li><strong>Social Media Marketing</strong> 0.377</li>
<li><strong>San Francisco</strong> 0.376</li>
<li><strong>Stealth Startups</strong> 0.362</li>
<li><strong>Yahoo!</strong> 0.36</li>
<li><strong>Cascading Style Sheets</strong> 0.359</li>
<li><strong>Angel Investors</strong> 0.355</li>
<li><strong>UX Design</strong> 0.35</li>
<li><strong>StarCraft</strong> 0.348</li>
<li><strong>Los Angeles Lakers</strong> 0.347</li>
<li><strong>Mountain View</strong> 0.345</li>
<li><strong>How I Met Your Mother (TV series)</strong> 0.338</li>
<li><strong>Google+</strong> 0.337</li>
<li><strong>Ruby on Rails</strong> 0.333</li>
<li><strong>Reading</strong> 0.333</li>
<li><strong>Social Media</strong> 0.326</li>
<li><strong>China</strong> 0.322</li>
<li><strong>Palantir Technologies</strong> 0.319</li>
<li><strong>Facebook Platform</strong> 0.315</li>
<li><strong>Basketball</strong> 0.315</li>
<li><strong>Education</strong> 0.314</li>
<li><strong>Business Development</strong> 0.312</li>
<li><strong>Online & Mobile Payments</strong> 0.305</li>
<li><strong>Restaurants in San Francisco</strong> 0.302</li>
<li><strong>Technology Companies</strong> 0.302</li>
<li><strong>Seth Godin</strong> 0.3</li>
</ul>
<h2>Caltech</h2>
<ul>
<li><strong>Pasadena</strong> 0.969</li>
<li><strong>Chess</strong> 0.748</li>
<li><strong>Table Tennis</strong> 0.671</li>
<li><strong>UCLA</strong> 0.67</li>
<li><strong>MacBook Pro</strong> 0.618</li>
<li><strong>Physics</strong> 0.618</li>
<li><strong>Haskell</strong> 0.582</li>
<li><strong>Los Angeles</strong> 0.58</li>
<li><strong>Electrical Engineering</strong> 0.567</li>
<li><strong>Star Trek (movie</strong> 0.561</li>
<li><strong>Disruptive Technology</strong> 0.545</li>
<li><strong>Science</strong> 0.53</li>
<li><strong>Biology</strong> 0.526</li>
<li><strong>Quantum Mechanics</strong> 0.521</li>
<li><strong>LaTeX</strong> 0.514</li>
<li><strong>Mathematics</strong> 0.488</li>
<li><strong>xkcd</strong> 0.488</li>
<li><strong>Genetics & Heredity</strong> 0.487</li>
<li><strong>Chemistry</strong> 0.47</li>
<li><strong>Medicine & Healthcare</strong> 0.448</li>
<li><strong>Poker</strong> 0.445</li>
<li><strong>C++ (programming language)</strong> 0.442</li>
<li><strong>Data Structures</strong> 0.434</li>
<li><strong>Emacs</strong> 0.428</li>
<li><strong>MongoDB</strong> 0.423</li>
<li><strong>Neuroscience</strong> 0.404</li>
<li><strong>Science Fiction</strong> 0.4</li>
<li><strong>Mac OS X</strong> 0.394</li>
<li><strong>Board Games</strong> 0.387</li>
<li><strong>Computers</strong> 0.386</li>
<li><strong>Research</strong> 0.385</li>
<li><strong>Finance</strong> 0.385</li>
<li><strong>The Future</strong> 0.379</li>
<li><strong>Linux</strong> 0.378</li>
<li><strong>The Colbert Report</strong> 0.376</li>
<li><strong>The Beatles</strong> 0.374</li>
<li><strong>The Onion</strong> 0.365</li>
<li><strong>Ruby</strong> 0.363</li>
<li><strong>Cars & Automobiles</strong> 0.361</li>
<li><strong>Quantitative Finance</strong> 0.359</li>
<li><strong>Academia</strong> 0.359</li>
<li><strong>Law</strong> 0.355</li>
<li><strong>Cooking</strong> 0.354</li>
<li><strong>Psychology</strong> 0.349</li>
<li><strong>Eminem</strong> 0.347</li>
<li><strong>Football (Soccer)</strong> 0.346</li>
<li><strong>Computer Programming</strong> 0.343</li>
<li><strong>Algorithms</strong> 0.343</li>
<li><strong>Evolutionary Biology</strong> 0.337</li>
<li><strong>Behavioral Economics</strong> 0.335</li>
<li><strong>California</strong> 0.329</li>
<li><strong>Machine Learning</strong> 0.326</li>
<li><strong>Futurama</strong> 0.324</li>
<li><strong>Social Advice</strong> 0.324</li>
<li><strong>StarCraft II</strong> 0.319</li>
<li><strong>Job Interview Questions</strong> 0.318</li>
<li><strong>Game Theory</strong> 0.316</li>
<li><strong>This American Life</strong> 0.315</li>
<li><strong>Economics</strong> 0.314</li>
<li><strong>Vim</strong> 0.31</li>
<li><strong>Graduate School</strong> 0.309</li>
<li><strong>Git (revision control)</strong> 0.306</li>
<li><strong>Computer Science</strong> 0.303</li>
</ul>
<p>What do we see?</p>
<ul>
<li>First, in a nice validation of this approach, we find that each school is interested in exactly the <strong>locations</strong> we’d expect: Caltech is interested in <em>Pasadena</em> and <em>Los Angeles</em>; MIT and Harvard are both interested in <em>Boston</em> and <em>Cambridge</em> (Harvard is interested in <em>New York City</em> as well); Stanford is interested in <em>Palo Alto</em>, <em>Northern California</em>, and <em>San Francisco Bay Area</em>; and Berkeley is interested in <em>Berkeley</em>, <em>San Francisco</em>, and <em>Mountain View</em>.</li>
<li>More interestingly, let’s look at where each school likes to <strong>eat</strong>. Stereotypically, we expect Harvard, Stanford, and Berkeley students to be more outgoing and social, and MIT and Caltech students to be more introverted. This is indeed what we find:
<ul>
<li>Harvard follows <em>Food in New York City</em>; Stanford follows <em>Food & Restaurants in Palo Alto</em>, <em>Restaurants in San Francisco</em>, and <em>Food in San Francisco</em>; and Berkeley follows <em>Restaurants in San Francisco</em> and <em>In-N-Out Burger</em>. In other words, Harvard, Stanford, and Berkeley love eating out.</li>
<li>Caltech, on the other hand, loves <em>Cooking</em>, and MIT loves <em>Healthy Eating</em> – both signs, perhaps, of a preference for eating in.</li>
</ul>
</li>
<li>And what does each university use to quench their <strong>thirst</strong>? Harvard students like to drink <em>wine</em> (classy!), while Stanford students prefer <em>beer</em> (the social drink of choice).</li>
<li>What about <strong>sports teams</strong>? MIT and Caltech couldn’t care less, though Harvard follows the <em>Boston Red Sox</em>, Stanford follows the <em>San Francisco Giants</em> (as well as their own <em>Stanford Football</em> and <em>Stanford Cardinal</em>), and Berkeley follows the <em>Los Angeles Lakers</em> (and the <em>California Golden Bears</em>).</li>
<li>For <strong>sports</strong> themselves, MIT students like <em>skiing</em>; Stanford students like <em>general exercise</em>, <em>fitness</em>, and <em>sports</em>; Harvard students like <em>baseball</em>, <em>ultimate frisbee</em>, and <em>skiing</em>; and Berkeley students like <em>snowboarding</em>. Caltech, in a league of its own, enjoys <em>table tennis</em> and <em>chess</em>.</li>
<li>What does each school think of <strong>social</strong>? Caltech students look for Social <em>Advice</em>. Berkeley students are interested in Social <em>Media</em> and Social Media Marketing. MIT, on the more technical side, wants Social <em>Search</em>. Stanford students, predictably, love the whole spectrum of social offerings, from Social <em>Dance</em> and <em>The Social Network</em>, to Social <em>Psychology</em> and Social <em>Advice</em>. (Interestingly, Caltech and Stanford are both interested in Social Advice, though I wonder if it’s for slightly different reasons.)</li>
<li>What’s each school’s relationship with <strong>computers</strong>? Caltech students are interested in Computer <em>Science</em>, MIT hackers are interested in Computer <em>Security</em>, and Stanford students are interested in Human-Computer <em>Interaction</em>.</li>
<li>Digging into the <strong>MIT vs. Caltech</strong> divide a little, we see that Caltech students really are more interested in the pure sciences (<em>Physics, Science, Biology, Quantum Mechanics, Mathematics, Chemistry</em>, etc.), while MIT students are more on the applied and engineering sides (<em>Mechanical Engineering, Engineering, Distributed Databases, Cryptography, Computer Security, Biotechnology, Operating Systems, Compilers</em>, etc.).</li>
<li>Regarding <strong>programming languages</strong>, Caltech students love <em>Haskell</em> (hardcore purity!), while MIT students love <em>Lisp</em>.</li>
<li>What does each school like to <strong>read</strong>, both offline and online? Caltech loves <em>science fiction</em>, <em>xkcd</em>, and <em>The Onion</em>; MIT likes <em>Hacker News</em>; Harvard loves journals, newspapers, and magazines (<em>Huffington Post</em>, the <em><a href="http://stuffwhitepeoplelike.com/2008/01/31/45-the-sunday-new-york-times/">New York Times</a></em>, <em>Fortune, Wall Street Journal, the New Yorker, the Economist</em>, and so on); and Stanford likes <em>TechMeme</em>.</li>
<li>What <strong>movies and television shows</strong> does each school like to watch? Caltech likes <em>Star Trek</em>, the <em>Colbert Report</em>, and <em>Futurama</em>. MIT likes <em>Fight Club</em> (I don’t know what this has to do with MIT, though I will note that on my first day as a freshman in a new dorm, Fight Club was precisely the movie we all went to a lecture hall to see). Stanford likes <em>The Social Network</em> and <em>Inception</em>. Harvard, rather fittingly, likes <em><a href="http://stuffwhitepeoplelike.com/2009/03/11/123-mad-men/">Mad Men</a></em> and <em><a href="http://stuffwhitepeoplelike.com/2010/09/08/134-the-ted-conference/">Ted Talks</a></em>.</li>
<li>Let’s look at the <strong>startups</strong> each school follows. MIT, of course, likes <em>Ksplice</em>. Berkeley likes <em>Yelp</em> and <em>Groupon</em>. Stanford likes just about every startup under the sun (<em>Instagram, Flipboard, Tumblr, Path, Color Labs</em>, etc.). And Harvard, that bastion of hard-won influence and prestige? To the surprise of precisely no one, Harvard enjoys <em>Klout</em>.</li>
</ul>
<p>Let’s end with a summarized view of each school:</p>
<ul>
<li><strong>Caltech</strong> is very much into the sciences (<em>Physics, Biology, Quantum Mechanics, Mathematics</em>, etc.), as well as many pretty nerdy topics (<em>Star Trek, Science Fiction, xkcd, Futurama, Starcraft II</em>, etc.).</li>
<li><strong>MIT</strong> is dominated by everything engineering and tech.</li>
<li><strong>Stanford</strong> loves relationships (<em>interpersonal relationships, people skills, love, network effects, sex, etiquette, dating and relationships, romance</em>), health and appearance (<em>fashion, fitness, nutrition, happiness</em>), and startups (<em>Instagram, Flipboard, Path, Color Labs</em>, etc.).</li>
<li><strong>Berkeley</strong>, sadly, is perhaps too large and diverse for an overall characterization.</li>
<li><strong>Harvard</strong> students are fascinated by famous figures (<em>Jimmy Fallon, Oprah Winfrey, Invaka Trump, Dalai Lama, David Lynch, Al Gore, Bill Gates, Barack Obama</em>), and by prestigious newspapers, journals, and magazines (<em>Fortune, the New York Times, the Wall Street Journal, the Economist</em>, and so on). Other very fitting interests include <em><a href="http://stuffwhitepeoplelike.com/2008/01/21/12-non-profit-organizations/">Kiva</a>, <a href="http://stuffwhitepeoplelike.com/2008/09/01/108-appearing-to-enjoy-classical-music/">classical music</a></em>, and <em><a href="http://www.vanityfair.com/online/daily/2008/06/coldplay">Coldplay</a></em>.</li>
</ul>
<p>*I pulled about 400 followers from each school, and added a couple filters, to try to ensure that followers were actual attendees of the schools rather than general people simply interested in them. Topics are sorted using a naive Bayes score and filtered to have at least 5 counts. Also, a word of warning: my dataset was fairly small and users on Quora are almost certainly not representative of their schools as a whole (though I tried to be rigorous with what I had).</p>
</div>
Information Transmission in a Social Network: Dissecting the Spread of a Quora Post2011-09-07T04:15:00+02:002011-09-07T04:15:00+02:00Edwin Chentag:blog.echen.me,2011-09-07:/2011/09/07/information-transmission-in-a-social-network-dissecting-the-spread-of-a-quora-post/
<div class="entry-content"><p><strong>tl;dr</strong> See <a href="http://www.youtube.com/watch?v=cZ4Ntg4jQHw">this movie visualization</a> for a case study on how a post propagates through Quora.</p>
<p>How does information spread through a network? Much of Quora’s appeal, after all, lies in its social graph – and when you’ve got a network of users, all broadcasting their activities to …</p></div>
<div class="entry-content"><p><strong>tl;dr</strong> See <a href="http://www.youtube.com/watch?v=cZ4Ntg4jQHw">this movie visualization</a> for a case study on how a post propagates through Quora.</p>
<p>How does information spread through a network? Much of Quora’s appeal, after all, lies in its social graph – and when you’ve got a network of users, all broadcasting their activities to their neighbors, information can cascade in multiple ways. How do these social designs affect which users see what?</p>
<p>Think, for example, of what happens when your kid learns a new slang word at school. He doesn’t confine his use of the word to McKinley Elementary’s particular boundaries, between the times of 9-3pm – he introduces it to his friends from other schools at soccer practice as well. A couple months later, he even says it at home for the first time; you like the word so much, you then start using it at work. Eventually, Justin Bieber uses the word in a song, at which point the word’s popularity really starts to explode.</p>
<p>So how does information propagate through a social network? What types of people does an answer on Quora reach, and how does it reach them? (Do users discover new answers individually, or are hubs of connectors more key?) How does the activity of a post on Quora rise and fall? (Submissions on other sites have limited lifetimes, fading into obscurity soon after an initial spike; how does that change when users are connected and every upvote can revive a post for someone else’s eyes?)</p>
<p>(I looked at Quora since I had some data from there already available, but I hope the lessons should be fairly applicable in general, to other social networks like Facebook, Twitter, and LinkedIn as well.)</p>
<p>To give an initial answer to some of these questions, I dug into one of my more popular posts, on <a href="http://www.quora.com/Random-Forests/How-do-random-forests-work-in-laymans-terms">a layman’s introduction to random forests</a>.</p>
<h1>Users, Topics</h1>
<p>Before looking deeper into the voting dynamics of the post, let’s first get some background on what kinds of users the answer reached.</p>
<p>Here’s a graph of the topics that question upvoters follow. (Each node is a topic, and every time upvoter X follows both topics A and B, I add an edge between A and B.)</p>
<p><a href="http://i.imgur.com/SYaWo7V.png"><img src="http://i.imgur.com/SYaWo7V.png" alt="Upvoters' Topics - Unlabeled" /></a></p>
<p><a href="http://i.imgur.com/A4Z1KF7.png"><img src="http://i.imgur.com/A4Z1KF7.png" alt="Upvoters' Topics - Labeled" /></a></p>
<p>We can see from the graph that upvoters tend to be interested in three kinds of topics:</p>
<ul>
<li><strong>Machine learning and other technical matters</strong> (the green cluster): Classification, Data Mining, Big Data, Information Retrieval, Analytics, Probability, Support Vector Machines, R, Data Science, …</li>
<li><strong>Startups/Silicon Valley</strong> (the red cluster): Facebook, Lean Startups, Investing, Seed Funding, Angel Investing, Technology Trends, Product Managment, Silicon Valley Mergers and Acquisitions, Asana, Social Games, Quora, Mark Zuckerberg, User Experience, Founders and Entrepreneurs, …</li>
<li><strong>General Intellectual Topics</strong> (the purple cluster): TED, Science, Book Recommendations, Philosophy, Politics, Self-Improvement, Travel, Life Hacks, …</li>
</ul>
<p>Also, here’s the network of the upvoters themselves (there’s an edge between users A and B if A follows B):</p>
<p><a href="http://i.imgur.com/JuqybXw.png"><img src="http://i.imgur.com/JuqybXw.png" alt="Upvote Network - Unlabeled" /></a></p>
<p><a href="http://i.imgur.com/0E2ekqO.png"><img src="http://i.imgur.com/0E2ekqO.png" alt="Upvote Network - Labeled" /></a></p>
<p>We can see three main clusters of users:</p>
<ul>
<li>A large group in <strong>green</strong> centered around a lot of power users and Quora employees.</li>
<li>A machine learning group of folks in <strong>orange</strong> centered around people like Oliver Grisel, Christian Langreiter, and Joseph Turian.</li>
<li>A group of people following me, in <strong>purple</strong>.</li>
<li>Plus some smaller clusters in blue and yellow. (There were also a bunch of isolated users, connected to no one, that I filtered out of the picture.)</li>
</ul>
<p>Digging into how these topic and user graphs are related:</p>
<ul>
<li>The orange cluster of users is more heavily into machine learning: 79% of users in that cluster follow more green topics (machine learning and technical topics) than red and purple topics (startups and general intellectual matters).</li>
<li>The green cluster of users is reversed: 77% of users follow more of the red and purple clusters of topics (on startups and general intellectual matters) than machine learning and technical topics.</li>
</ul>
<p>More interestingly, though, we can ask: how do the connections between upvoters relate to the way the post spread?</p>
<h1>Social Voting Dynamics</h1>
<p>So let’s take a look. Here’s a visualization I made of upvotes on my answer across time (click <a href="http://www.youtube.com/watch?v=cZ4Ntg4jQHw">here</a> for a larger view).</p>
<iframe width="640" height="510" src="http://www.youtube.com/embed/cZ4Ntg4jQHw " frameborder="0" allowfullscreen></iframe>
<p></p>
<p>To represent the social dynamics of these upvotes, I drew an edge from user A to user B if user A transmitted the post to user B through an upvote. (Specifically, I drew an edge from Alice to Bob if Bob follows Alice and Bob’s upvote appeared within five days of Alice’s upvote; this is meant to simulate the idea that Alice was the key intermediary between my post and Bob.)</p>
<p>Also,</p>
<ul>
<li>Green nodes are users with at least one upvote edge.</li>
<li>Blue nodes are users who follow at least one of the topics the post is categorized under (i.e., users who probably discovered the answer by themselves).</li>
<li>Red nodes are users with no connections and who do not follow any of the post’s topics (i.e, users whose path to the post remain mysterious).</li>
<li>Users increase in size when they produce more connections.</li>
</ul>
<p>Here’s a play-by-play of the video:</p>
<ul>
<li>On Feb 14 (the day I wrote the answer), there’s a flurry of activity.</li>
<li>A couple of days later, Tracy Chou gives an upvote, leading to another spike in activity.</li>
<li>Then all’s quiet until… bam! Alex Kamil leads to a surge of upvotes, and his upvote finds Ludi Rehak, who starts a small surge of her own. They’re quickly followed by Christian Langreiter, who starts a small revolution among a bunch of machine learning folks a couple days later.</li>
<li>Then all is pretty calm again, until a couple months later when… bam! Aditya Sengupta brings in a smashing of his own followers, and his upvote makes its way to Marc Bodnick, who sets off a veritable storm of activity.</li>
</ul>
<p>(Already we can see some relationships between the graph of user connections and the way the post propagated. Many of the users from the orange cluster, for example, come from Alex Kamil and Christian Langreiter’s upvotes, and many of the users from the green cluster come from Aditya Sengupta and Marc Bodnick’s upvotes. What’s interesting, though, is, why didn’t the cluster of green users appear all at once, like the orange cluster did? People like Kah Seng Tay, Tracy Chou, Venkatesh Rao, and Chad Little upvoted the answer pretty early on, but it wasn’t until Aditya Sengupta’s upvote a couple months later that people like Marc Bodnick, Edmond Lau, and many of the other green users (who do indeed follow that first set of folks) discovered the answer. Did the post simply get lost in users’ feeds the first time around? Was the post perhaps ignored until it received enough upvotes to be considered worth reading? Are some users’ upvotes just trusted more than others’?)</p>
<p>For another view of the upvote dynamics, here’s a static visualization, where we can again easily see the clusters of activity:</p>
<p><a href="http://i.imgur.com/jmOacPR.png"><img src="http://i.imgur.com/jmOacPR.png" alt="Upvote Temporal Clusters" /></a></p>
<h1>Fin</h1>
<p>There are still many questions it would be interesting to look at; for example,</p>
<ul>
<li>What differentiates users who sparked spikes of activity from users who didn’t? I don’t believe it’s simply number of followers, as many well-connected upvoters did <em>not</em> lead to cascades of shares. Does authority matter?</li>
<li>How far can a post reach? Clearly, the post reached people more than one degree of separation away from me (where one degree of separation is a follower); what does the distribution of degrees look like? Is there any relationship between degree of separation and time of upvote?</li>
<li>What can we say about the people who started following me after reading my answer? Are they fewer degrees of separation away? Are they more interested in machine learning? Have they upvoted any of my answers before? (Perhaps there’s a certain “threshold” of interestingness people need to overflow before they’re considered acceptable followees.)</li>
</ul>
<p>But to summarize a bit what we’ve seen so far, here are some statistics on the role the social graph played in spreading the post:</p>
<ul>
<li>There are 5 clusters of activity after the initial post, sparked both by power users and less-connected folks. In an interesting cascade of information, some of these sparks led to further spikes in activity as well (as when Aditya Sengupta’s upvote found its way to Marc Bodnick, who set off even more activity).</li>
<li>35% of users made their way to my answer because of someone else’s upvote.</li>
<li>Through these connections, the post reached a fair variety of users: 32% of upvoters don’t even follow any of the post’s topics.</li>
<li>77% of upvotes came from users over two weeks <em>after</em> my answer appeared.</li>
<li>If we look only at the upvoters who follow at least one of the post’s topics, 33% didn’t see my answer until someone else showed it to them. In other words, a full one-third of people who presumably would have been interested in my post anyways only found it because of their social network.</li>
</ul>
<p>So it looks like the social graph played quite a large part in the post’s propagation, and I’ll end with a big shoutout to Stormy Shippy, who provided an awesome set of scripts I used to collect a lot of this data.</p>
</div>
Introduction to Latent Dirichlet Allocation2011-08-22T04:15:00+02:002011-08-22T04:15:00+02:00Edwin Chentag:blog.echen.me,2011-08-22:/2011/08/22/introduction-to-latent-dirichlet-allocation/
<div class="entry-content"><h1>Introduction</h1>
<p>Suppose you have the following set of sentences:</p>
<ul>
<li>I like to eat broccoli and bananas.</li>
<li>I ate a banana and spinach smoothie for breakfast.</li>
<li>Chinchillas and kittens are cute.</li>
<li>My sister adopted a kitten yesterday.</li>
<li>Look at this cute hamster munching on a piece of broccoli.</li>
</ul>
<p>What is latent …</p></div>
<div class="entry-content"><h1>Introduction</h1>
<p>Suppose you have the following set of sentences:</p>
<ul>
<li>I like to eat broccoli and bananas.</li>
<li>I ate a banana and spinach smoothie for breakfast.</li>
<li>Chinchillas and kittens are cute.</li>
<li>My sister adopted a kitten yesterday.</li>
<li>Look at this cute hamster munching on a piece of broccoli.</li>
</ul>
<p>What is latent Dirichlet allocation? It’s a way of automatically discovering <strong>topics</strong> that these sentences contain. For example, given these sentences and asked for 2 topics, LDA might produce something like</p>
<ul>
<li><strong>Sentences 1 and 2</strong>: 100% Topic A</li>
<li><strong>Sentences 3 and 4</strong>: 100% Topic B</li>
<li><strong>Sentence 5</strong>: 60% Topic A, 40% Topic B</li>
<li><strong>Topic A</strong>: 30% broccoli, 15% bananas, 10% breakfast, 10% munching, … (at which point, you could interpret topic A to be about food)</li>
<li><strong>Topic B</strong>: 20% chinchillas, 20% kittens, 20% cute, 15% hamster, … (at which point, you could interpret topic B to be about cute animals)</li>
</ul>
<p>The question, of course, is: how does LDA perform this discovery?</p>
<h1>LDA Model</h1>
<p>In more detail, LDA represents documents as <strong>mixtures of topics</strong> that spit out words with certain probabilities. It assumes that documents are produced in the following fashion: when writing each document, you</p>
<ul>
<li>Decide on the number of words N the document will have (say, according to a Poisson distribution).</li>
<li>Choose a topic mixture for the document (according to a Dirichlet distribution over a fixed set of K topics). For example, assuming that we have the two food and cute animal topics above, you might choose the document to consist of 1/3 food and 2/3 cute animals.</li>
<li>Generate each word w_i in the document by:
<ul>
<li>First picking a topic (according to the multinomial distribution that you sampled above; for example, you might pick the food topic with 1/3 probability and the cute animals topic with 2/3 probability).</li>
<li>Using the topic to generate the word itself (according to the topic’s multinomial distribution). For example, if we selected the food topic, we might generate the word “broccoli” with 30% probability, “bananas” with 15% probability, and so on.</li>
</ul>
</li>
</ul>
<p>Assuming this generative model for a collection of documents, LDA then tries to backtrack from the documents to find a set of topics that are likely to have generated the collection.</p>
<h2>Example</h2>
<p>Let’s make an example. According to the above process, when generating some particular document D, you might</p>
<ul>
<li>Pick 5 to be the number of words in D.</li>
<li>Decide that D will be 1/2 about food and 1/2 about cute animals.</li>
<li>Pick the first word to come from the food topic, which then gives you the word “broccoli”.</li>
<li>Pick the second word to come from the cute animals topic, which gives you “panda”.</li>
<li>Pick the third word to come from the cute animals topic, giving you “adorable”.</li>
<li>Pick the fourth word to come from the food topic, giving you “cherries”.</li>
<li>Pick the fifth word to come from the food topic, giving you “eating”.</li>
</ul>
<p>So the document generated under the LDA model will be “broccoli panda adorable cherries eating” (note that LDA is a bag-of-words model).</p>
<h1>Learning</h1>
<p>So now suppose you have a set of documents. You’ve chosen some fixed number of K topics to discover, and want to use LDA to learn the topic representation of each document and the words associated to each topic. How do you do this? One way (known as collapsed Gibbs sampling) is the following:</p>
<ul>
<li>Go through each document, and randomly assign each word in the document to one of the K topics.</li>
<li>Notice that this random assignment already gives you both topic representations of all the documents and word distributions of all the topics (albeit not very good ones).</li>
<li>So to improve on them, for each document d…
<ul>
<li>Go through each word w in d…
<ul>
<li>And for each topic t, compute two things: 1) p(topic t | document d) = the proportion of words in document d that are currently assigned to topic t, and 2) p(word w | topic t) = the proportion of assignments to topic t over all documents that come from this word w. Reassign w a new topic, where we choose topic t with probability p(topic t | document d) * p(word w | topic t) (according to our generative model, this is essentially the probability that topic t generated word w, so it makes sense that we resample the current word’s topic with this probability). (Also, I’m glossing over a couple of things here, in particular the use of priors/pseudocounts in these probabilities.)</li>
<li>In other words, in this step, we’re assuming that all topic assignments except for the current word in question are correct, and then updating the assignment of the current word using our model of how documents are generated.</li>
</ul>
</li>
</ul>
</li>
<li>After repeating the previous step a large number of times, you’ll eventually reach a roughly steady state where your assignments are pretty good. So use these assignments to estimate the topic mixtures of each document (by counting the proportion of words assigned to each topic within that document) and the words associated to each topic (by counting the proportion of words assigned to each topic overall).</li>
</ul>
<h1>Layman’s Explanation</h1>
<p>In case the discussion above was a little eye-glazing, here’s another way to look at LDA in a different domain.</p>
<p>Suppose you’ve just moved to a new city. You’re a hipster and an anime fan, so you want to know where the other hipsters and anime geeks tend to hang out. Of course, as a hipster, you know you can’t just <em>ask</em>, so what do you do?</p>
<p>Here’s the scenario: you scope out a bunch of different establishments (<strong>documents</strong>) across town, making note of the people (<strong>words</strong>) hanging out in each of them (e.g., Alice hangs out at the mall and at the park, Bob hangs out at the movie theater and the park, and so on). Crucially, you don’t know the typical interest groups (<strong>topics</strong>) of each establishment, nor do you know the different interests of each person.</p>
<p>So you pick some number K of categories to learn (i.e., you want to learn the K most important kinds of categories people fall into), and start by making a guess as to why you see people where you do. For example, you initially guess that Alice is at the mall because people with interests in X like to hang out there; when you see her at the park, you guess it’s because her friends with interests in Y like to hang out there; when you see Bob at the movie theater, you randomly guess it’s because the Z people in this city really like to watch movies; and so on.</p>
<p>Of course, your random guesses are very likely to be incorrect (they’re random guesses, after all!), so you want to improve on them. One way of doing so is to:</p>
<ul>
<li>Pick a place and a person (e.g., Alice at the mall).</li>
<li>Why is Alice likely to be at the mall? Probably because other people at the mall with the same interests sent her a message telling her to come.</li>
<li>In other words, the more people with interests in X there are at the mall and the stronger Alice is associated with interest X (at all the other places she goes to), the more likely it is that Alice is at the mall because of interest X.</li>
<li>So make a new guess as to why Alice is at the mall, choosing an interest with some probability according to how likely you think it is.</li>
</ul>
<p>Go through each place and person over and over again. Your guesses keep getting better and better (after all, if you notice that lots of geeks hang out at the bookstore, and you suspect that Alice is pretty geeky herself, then it’s a good bet that Alice is at the bookstore because her geek friends told her to go there; and now that you have a better idea of why Alice is probably at the bookstore, you can use this knowledge in turn to improve your guesses as to why everyone else is where they are), and eventually you can stop updating. Then take a snapshot (or multiple snapshots) of your guesses, and use it to get all the information you want:</p>
<ul>
<li>For each category, you can count the people assigned to that category to figure out what people have this particular interest. By looking at the people themselves, you can interpret the category as well (e.g., if category X contains lots of tall people wearing jerseys and carrying around basketballs, you might interpret X as the “basketball players” group).</li>
<li>For each place P and interest category C, you can compute the proportions of people at P because of C (under the current set of assignments), and these give you a representation of P. For example, you might learn that the people who hang out at Barnes & Noble consist of 10% hipsters, 50% anime fans, 10% jocks, and 30% college students.</li>
</ul>
<h1>Real-World Example</h1>
<p>Finally, I applied LDA to a set of Sarah Palin’s emails a little while ago (see <a href="http://blog.echen.me/2011/06/27/topic-modeling-the-sarah-palin-emails/">here</a> for the blog post, or <a href="http://sarah-palin.heroku.com/">here</a> for an app that allows you to browse through the emails by the LDA-learned categories), so let’s give a brief recap. Here are some of the topics that the algorithm learned:</p>
<ul>
<li><strong>Trig/Family/Inspiration</strong>: family, web, mail, god, son, from, congratulations, children, life, child, down, trig, baby, birth, love, you, syndrome, very, special, bless, old, husband, years, thank, best, …</li>
<li><strong>Wildlife/BP Corrosion</strong>: game, fish, moose, wildlife, hunting, bears, polar, bear, subsistence, management, area, board, hunt, wolves, control, department, year, use, wolf, habitat, hunters, caribou, program, denby, fishing, …</li>
<li><strong>Energy/Fuel/Oil/Mining:</strong> energy, fuel, costs, oil, alaskans, prices, cost, nome, now, high, being, home, public, power, mine, crisis, price, resource, need, community, fairbanks, rebate, use, mining, villages, …</li>
<li><strong>Gas</strong>: gas, oil, pipeline, agia, project, natural, north, producers, companies, tax, company, energy, development, slope, production, resources, line, gasline, transcanada, said, billion, plan, administration, million, industry, …</li>
<li><strong>Education/Waste</strong>: school, waste, education, students, schools, million, read, email, market, policy, student, year, high, news, states, program, first, report, business, management, bulletin, information, reports, 2008, quarter, …</li>
<li><strong>Presidential Campaign/Elections</strong>: mail, web, from, thank, you, box, mccain, sarah, very, good, great, john, hope, president, sincerely, wasilla, work, keep, make, add, family, republican, support, doing, p.o, …</li>
</ul>
<p>Here’s an example of an email which fell 99% into the Trig/Family/Inspiration category (particularly representative words are highlighted in blue):</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/palin-browser/trig-email.png"><img src="http://dl.dropbox.com/u/10506/blog/palin-browser/trig-email.png" alt="Trig Email" /></a></p>
<p>And here’s an excerpt from an email which fell 10% into the Presidential Campaign/Election category (in red) and 90% into the Wildlife/BP Corrosion category (in green):</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/palin-browser/wildlife-presidency-email.png"><img src="http://dl.dropbox.com/u/10506/blog/palin-browser/wildlife-presidency-email.png" alt="Wildlife-Presidency Email" /></a></p>
</div>
Introduction to Restricted Boltzmann Machines2011-07-18T00:00:00+02:002011-07-18T00:00:00+02:00Edwin Chentag:blog.echen.me,2011-07-18:/2011/07/18/introduction-to-restricted-boltzmann-machines/<p>Suppose you ask a bunch of users to rate a set of movies on a 0-100 scale. In classical factor analysis, you could then try to explain each movie and user in terms of a set of latent <strong>factors</strong>. For example, movies like Star Wars and Lord of the Rings …</p><p>Suppose you ask a bunch of users to rate a set of movies on a 0-100 scale. In classical factor analysis, you could then try to explain each movie and user in terms of a set of latent <strong>factors</strong>. For example, movies like Star Wars and Lord of the Rings might have strong associations with a latent science fiction and fantasy factor, and users who like Wall-E and Toy Story might have strong associations with a latent Pixar factor.</p>
<p>Restricted Boltzmann Machines essentially perform a <em>binary</em> version of factor analysis. (This is one way of thinking about RBMs; there are, of course, others, and lots of different ways to use RBMs, but I’ll adopt this approach for this post.) Instead of users rating a set of movies on a continuous scale, they simply tell you whether they like a movie or not, and the RBM will try to discover latent factors that can explain the activation of these movie choices.</p>
<p>More technically, a Restricted Boltzmann Machine is a <strong>stochastic neural network</strong> (neural network meaning we have neuron-like units whose binary activations depend on the neighbors they’re connected to; stochastic meaning these activations have a probabilistic element) consisting of:</p>
<ul>
<li>One layer of <strong>visible units</strong> (users’ movie preferences whose states we know and set);</li>
<li>One layer of <strong>hidden units</strong> (the latent factors we try to learn); and</li>
<li>A bias unit (whose state is always on, and is a way of adjusting for the different inherent popularities of each movie).</li>
</ul>
<p>Furthermore, each visible unit is connected to all the hidden units (this connection is undirected, so each hidden unit is also connected to all the visible units), and the bias unit is connected to all the visible units and all the hidden units. To make learning easier, we restrict the network so that no visible unit is connected to any other visible unit and no hidden unit is connected to any other hidden unit.</p>
<p>For example, suppose we have a set of six movies (Harry Potter, Avatar, LOTR 3, Gladiator, Titanic, and Glitter) and we ask users to tell us which ones they want to watch. If we want to learn two latent units underlying movie preferences – for example, two natural groups in our set of six movies appear to be SF/fantasy (containing Harry Potter, Avatar, and LOTR 3) and Oscar winners (containing LOTR 3, Gladiator, and Titanic), so we might hope that our latent units will correspond to these categories – then our RBM would look like the following:</p>
<p><img alt="RBM Example" src="http://i.imgur.com/sadnLks.png"></p>
<p>(Note the resemblance to a factor analysis graphical model.)</p>
<h1>State Activation</h1>
<p>Restricted Boltzmann Machines, and neural networks in general, work by updating the states of some neurons given the states of others, so let’s talk about how the states of individual units change. Assuming we know the connection weights in our RBM (we’ll explain how to learn these below), to update the state of unit <span class="math">\(i\)</span>:</p>
<ul>
<li>Compute the <strong>activation energy</strong> <span class="math">\(a_i = \sum_j w_{ij} x_j\)</span> of unit <span class="math">\(i\)</span>, where the sum runs over all units <span class="math">\(j\)</span> that unit <span class="math">\(i\)</span> is connected to, <span class="math">\(w_{ij}\)</span> is the weight of the connection between <span class="math">\(i\)</span> and <span class="math">\(j\)</span>, and <span class="math">\(x_j\)</span> is the 0 or 1 state of unit <span class="math">\(j\)</span>. In other words, all of unit <span class="math">\(i\)</span>’s neighbors send it a message, and we compute the sum of all these messages.</li>
<li>Let <span class="math">\(p_i = \sigma(a_i)\)</span>, where <span class="math">\(\sigma(x) = 1/(1 + exp(-x))\)</span> is the logistic function. Note that <span class="math">\(p_i\)</span> is close to 1 for large positive activation energies, and <span class="math">\(p_i\)</span> is close to 0 for negative activation energies.</li>
<li>We then turn unit <span class="math">\(i\)</span> on with probability <span class="math">\(p_i\)</span>, and turn it off with probability <span class="math">\(1 - p_i\)</span>.</li>
<li>(In layman’s terms, units that are positively connected to each other try to get each other to share the same state (i.e., be both on or off), while units that are negatively connected to each other are enemies that prefer to be in different states.)</li>
</ul>
<p>For example, let’s suppose our two hidden units really do correspond to SF/fantasy and Oscar winners.</p>
<ul>
<li>If Alice has told us her six binary preferences on our set of movies, we could then ask our RBM which of the hidden units her preferences activate (i.e., ask the RBM to explain her preferences in terms of latent factors). So the six movies send messages to the hidden units, telling them to update themselves. (Note that even if Alice has declared she wants to watch Harry Potter, Avatar, and LOTR 3, this doesn’t guarantee that the SF/fantasy hidden unit will turn on, but only that it will turn on with high probability. This makes a bit of sense: in the real world, Alice wanting to watch all three of those movies makes us highly suspect she likes SF/fantasy in general, but there’s a small chance she wants to watch them for other reasons. Thus, the RBM allows us to generate models of people in the messy, real world.)</li>
<li>Conversely, if we know that one person likes SF/fantasy (so that the SF/fantasy unit is on), we can then ask the RBM which of the movie units that hidden unit turns on (i.e., ask the RBM to generate a set of movie recommendations). So the hidden units send messages to the movie units, telling them to update their states. (Again, note that the SF/fantasy unit being on doesn’t guarantee that we’ll always recommend all three of Harry Potter, Avatar, and LOTR 3 because, hey, not everyone who likes science fiction liked Avatar.)</li>
</ul>
<h1>Learning Weights</h1>
<p>So how do we learn the connection weights in our network? Suppose we have a bunch of training examples, where each training example is a binary vector with six elements corresponding to a user’s movie preferences. Then for each epoch, do the following:</p>
<ul>
<li>Take a training example (a set of six movie preferences). Set the states of the visible units to these preferences.</li>
<li>Next, update the states of the hidden units using the logistic activation rule described above: for the <span class="math">\(j\)</span>th hidden unit, compute its activation energy <span class="math">\(a_j = \sum_i w_{ij} x_i\)</span>, and set <span class="math">\(x_j\)</span> to 1 with probability <span class="math">\(\sigma(a_j)\)</span> and to 0 with probability <span class="math">\(1 - \sigma(a_j)\)</span>. Then for each edge <span class="math">\(e_{ij}\)</span>, compute <span class="math">\(Positive(e_{ij}) = x_i * x_j\)</span> (i.e., for each pair of units, measure whether they’re both on).</li>
<li>Now <strong>reconstruct</strong> the visible units in a similar manner: for each visible unit, compute its activation energy <span class="math">\(a_i\)</span>, and update its state. (Note that this reconstruction may not match the original preferences.) Then update the hidden units again, and compute <span class="math">\(Negative(e_{ij}) = x_i * x_j\)</span> for each edge.</li>
<li>Update the weight of each edge <span class="math">\(e_{ij}\)</span> by setting <span class="math">\(w_{ij} = w_{ij} + L * (Positive(e_{ij}) - Negative(e_{ij}))\)</span>, where <span class="math">\(L\)</span> is a learning rate.</li>
<li>Repeat over all training examples.</li>
</ul>
<p>Continue until the network converges (i.e., the error between the training examples and their reconstructions falls below some threshold) or we reach some maximum number of epochs.</p>
<p>Why does this update rule make sense? Note that</p>
<ul>
<li>In the first phase, <span class="math">\(Positive(e_{ij})\)</span> measures the association between the <span class="math">\(i\)</span>th and <span class="math">\(j\)</span>th unit that we want the network to learn from our training examples;</li>
<li>In the “reconstruction” phase, where the RBM generates the states of visible units based on its hypotheses about the hidden units alone, <span class="math">\(Negative(e_{ij})\)</span> measures the association that the network itself generates (or “daydreams” about) when no units are fixed to training data.</li>
</ul>
<p>So by adding <span class="math">\(Positive(e_{ij}) - Negative(e_{ij})\)</span> to each edge weight, we’re helping the network’s daydreams better match the reality of our training examples.</p>
<p>(You may hear this update rule called <strong>contrastive divergence</strong>, which is basically a fancy term for “approximate gradient descent”.)</p>
<h1>Examples</h1>
<p>I wrote a <a href="https://github.com/echen/restricted-boltzmann-machines">simple RBM implementation</a> in Python (the code is heavily commented, so take a look if you’re still a little fuzzy on how everything works), so let’s use it to walk through some examples.</p>
<p>First, I trained the RBM using some fake data.</p>
<ul>
<li>Alice: (Harry Potter = 1, Avatar = 1, LOTR 3 = 1, Gladiator = 0, Titanic = 0, Glitter = 0). Big SF/fantasy fan.</li>
<li>Bob: (Harry Potter = 1, Avatar = 0, LOTR 3 = 1, Gladiator = 0, Titanic = 0, Glitter = 0). SF/fantasy fan, but doesn’t like Avatar.</li>
<li>Carol: (Harry Potter = 1, Avatar = 1, LOTR 3 = 1, Gladiator = 0, Titanic = 0, Glitter = 0). Big SF/fantasy fan.</li>
<li>David: (Harry Potter = 0, Avatar = 0, LOTR 3 = 1, Gladiator = 1, Titanic = 1, Glitter = 0). Big Oscar winners fan.</li>
<li>Eric: (Harry Potter = 0, Avatar = 0, LOTR 3 = 1, Gladiator = 1, Titanic = 1, Glitter = 0). Oscar winners fan, except for Titanic.</li>
<li>Fred: (Harry Potter = 0, Avatar = 0, LOTR 3 = 1, Gladiator = 1, Titanic = 1, Glitter = 0). Big Oscar winners fan.</li>
</ul>
<p>The network learned the following weights:</p>
<p><img alt="Weights" src="http://i.imgur.com/mbsjJmX.png"></p>
<p>Note that the first hidden unit seems to correspond to the Oscar winners, and the second hidden unit seems to correspond to the SF/fantasy movies, just as we were hoping.</p>
<p>What happens if we give the RBM a new user, George, who has (Harry Potter = 0, Avatar = 0, LOTR 3 = 0, Gladiator = 1, Titanic = 1, Glitter = 0) as his preferences? It turns the Oscar winners unit on (but not the SF/fantasy unit), correctly guessing that George probably likes movies that are Oscar winners.</p>
<p>What happens if we activate only the SF/fantasy unit, and run the RBM a bunch of different times? In my trials, it turned on Harry Potter, Avatar, and LOTR 3 three times; it turned on Avatar and LOTR 3, but not Harry Potter, once; and it turned on Harry Potter and LOTR 3, but not Avatar, twice. Note that, based on our training examples, these generated preferences do indeed match what we might expect real SF/fantasy fans want to watch.</p>
<h1>Modifications</h1>
<p>I tried to keep the connection-learning algorithm I described above pretty simple, so here are some modifications that often appear in practice:</p>
<ul>
<li>Above, <span class="math">\(Negative(e_{ij})\)</span> was determined by taking the product of the <span class="math">\(i\)</span>th and <span class="math">\(j\)</span>th units after reconstructing the visible units once and then updating the hidden units again. We could also take the product after some larger number of reconstructions (i.e., repeat updating the visible units, then the hidden units, then the visible units again, and so on); this is slower, but describes the network’s daydreams more accurately.</li>
<li>Instead of using <span class="math">\(Positive(e_{ij})=x_i * x_j\)</span>, where <span class="math">\(x_i\)</span> and <span class="math">\(x_j\)</span> are binary 0 or 1 states, we could also let <span class="math">\(x_i\)</span> and/or <span class="math">\(x_j\)</span> be activation probabilities. Similarly for <span class="math">\(Negative(e_{ij})\)</span>.</li>
<li>We could penalize larger edge weights, in order to get a sparser or more regularized model.</li>
<li>When updating edge weights, we could use a momentum factor: we would add to each edge a weighted sum of the current step as described above (i.e., <span class="math">\(L * (Positive(e_{ij}) - Negative(e_{ij})\)</span>) and the step previously taken.</li>
<li>Instead of using only one training example in each epoch, we could use batches of examples in each epoch, and only update the network’s weights after passing through all the examples in the batch. This can speed up the learning by taking advantage of fast matrix-multiplication algorithms.</li>
</ul>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Topic Modeling the Sarah Palin Emails2011-06-27T04:15:00+02:002011-06-27T04:15:00+02:00Edwin Chentag:blog.echen.me,2011-06-27:/2011/06/27/topic-modeling-the-sarah-palin-emails/
<div class="entry-content"><h1>LDA-based Email Browser</h1>
<p>Earlier this month, several thousand emails from Sarah Palin’s time as governor of Alaska were <a href="http://sunlightlabs.com/blog/2011/sarahs-inbox/">released</a>. The emails weren’t organized in any fashion, though, so to make them easier to browse, I’ve been working on some topic modeling (in particular, using latent Dirichlet allocation …</p></div>
<div class="entry-content"><h1>LDA-based Email Browser</h1>
<p>Earlier this month, several thousand emails from Sarah Palin’s time as governor of Alaska were <a href="http://sunlightlabs.com/blog/2011/sarahs-inbox/">released</a>. The emails weren’t organized in any fashion, though, so to make them easier to browse, I’ve been working on some topic modeling (in particular, using latent Dirichlet allocation) to separate the documents into different groups.</p>
<p>I threw up <a href="http://sarah-palin.heroku.com/">a simple demo app</a> to view the organized documents <a href="http://sarah-palin.heroku.com/">here</a>.</p>
<h1>What is Latent Dirichlet Allocation?</h1>
<p>Briefly, given a set of documents, LDA tries to learn the latent topics underlying the set. It represents each document as a mixture of topics (generated from a Dirichlet distribution), each of which emits words with a certain probability.</p>
<p>For example, given the sentence “I listened to Justin Bieber and Lady Gaga on the radio while driving around in my car”, an LDA model might represent this sentence as 75% about music (a topic which, say, emits the words <em>Bieber</em> with 10% probability, <em>Gaga</em> with 5% probability, <em>radio</em> with 1% probability, and so on) and 25% about cars (which might emit <em>driving</em> with 15% probability and <em>cars</em> with 10% probability).</p>
<p>If you’re familiar with latent semantic analysis, you can think of LDA as a generative version. (For a more in-depth explanation, I wrote an introduction to LDA <a href="http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/">here</a>.)</p>
<h1>Sarah Palin Email Topics</h1>
<p>Here’s a sample of the topics learnt by the model, as well as the top words for each topic. (Names, of course, are based on my own interpretation.)</p>
<ul>
<li><a href="http://sarah-palin.heroku.com/topics/24"><strong>Wildlife/BP Corrosion</strong></a>: game, fish, moose, wildlife, hunting, bears, polar, bear, subsistence, management, area, board, hunt, wolves, control, department, year, use, wolf, habitat, hunters, caribou, program, denby, fishing, …</li>
<li><a href="http://sarah-palin.heroku.com/topics/0"><strong>Energy/Fuel/Oil/Mining</strong></a>: energy, fuel, costs, oil, alaskans, prices, cost, nome, now, high, being, home, public, power, mine, crisis, price, resource, need, community, fairbanks, rebate, use, mining, villages, …</li>
<li><a href="http://sarah-palin.heroku.com/topics/19"><strong>Trig/Family/Inspiration</strong></a>: family, web, mail, god, son, from, congratulations, children, life, child, down, trig, baby, birth, love, you, syndrome, very, special, bless, old, husband, years, thank, best, …</li>
<li><a href="http://sarah-palin.heroku.com/topics/6"><strong>Gas</strong></a>: gas, oil, pipeline, agia, project, natural, north, producers, companies, tax, company, energy, development, slope, production, resources, line, gasline, transcanada, said, billion, plan, administration, million, industry, …</li>
<li><a href="http://sarah-palin.heroku.com/topics/12"><strong>Education/Waste</strong></a>: school, waste, education, students, schools, million, read, email, market, policy, student, year, high, news, states, program, first, report, business, management, bulletin, information, reports, 2008, quarter, …</li>
<li><a href="http://sarah-palin.heroku.com/topics/15"><strong>Presidential Campaign/Elections</strong></a>: mail, web, from, thank, you, box, mccain, sarah, very, good, great, john, hope, president, sincerely, wasilla, work, keep, make, add, family, republican, support, doing, p.o, …</li>
</ul>
<p>Here’s a sample email from the wildlife topic:</p>
<p><a href="http://sarah-palin.heroku.com/emails/6719"><img src="http://dl.dropbox.com/u/10506/blog/palin-browser/wildlife-email.png" alt="Wildlife Email" /></a></p>
<p>I also thought the classification for <a href="http://sarah-palin.heroku.com/emails/12900">this email</a> was really neat: the LDA model labeled it as 10% in the <a href="http://sarah-palin.heroku.com/topics/15">Presidential Campaign/Elections</a> topic and 90% in the <a href="http://sarah-palin.heroku.com/topics/24">Wildlife</a> topic, and it’s precisely a wildlife-based protest against Palin as a choice for VP:</p>
<p><a href="http://sarah-palin.heroku.com/emails/12900"><img src="http://dl.dropbox.com/u/10506/blog/palin-browser/wildlife-vp.png" alt="Wildlife-VP Protest" /></a></p>
<h1>Future Analysis</h1>
<p>In a future post, I’ll perhaps see if we can glean any interesting patterns from the email topics. For example, for a quick graph now, if we look at the percentage of emails in the <a href="http://sarah-palin.heroku.com/topics/19">Trig/Family/Inspiration topic</a> across time, we see that there’s a spike in April 2008 – exactly (and unsurprisingly) the month in which Trig was born.</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/palin-browser/trig-topic.png"><img src="http://dl.dropbox.com/u/10506/blog/palin-browser/trig-topic.png" alt="Trig" /></a></p>
</div>
Filtering for English Tweets: Unsupervised Language Detection on Twitter2011-05-01T04:15:00+02:002011-05-01T04:15:00+02:00Edwin Chentag:blog.echen.me,2011-05-01:/2011/05/01/unsupervised-language-detection-algorithms/
<div class="entry-content"><p>(See a demo <a href="http://babel-fett.heroku.com/">here</a>.)</p>
<p>While working on a Twitter sentiment analysis project, I ran into the problem of needing to filter out all non-English tweets. (Asking the Twitter API for English-only tweets doesn’t seem to work, as it nonetheless returns tweets in Spanish, Portuguese, Dutch, Russian, and a couple …</p></div>
<div class="entry-content"><p>(See a demo <a href="http://babel-fett.heroku.com/">here</a>.)</p>
<p>While working on a Twitter sentiment analysis project, I ran into the problem of needing to filter out all non-English tweets. (Asking the Twitter API for English-only tweets doesn’t seem to work, as it nonetheless returns tweets in Spanish, Portuguese, Dutch, Russian, and a couple other languages.)</p>
<p>Since I didn’t have any labeled data, I thought it would be fun to build an <strong>unsupervised</strong> language classifier. In particular, using an EM algorithm to build a naive Bayes model of English vs. non-English n-gram probabilities turned out to work quite well, so here’s a description.</p>
<h1>EM Algorithm</h1>
<p>Let’s recall the naive Bayes algorithm: given a tweet (a set of <em>character</em> n-grams), we estimate its language to be the language $L$ that maximizes</p>
<p>$$P(language = L | ngrams) \propto P(ngrams | language = L) P(language = L)$$</p>
<p>Thus, we need to estimate $P(ngram | language = L)$ and $P(language = L)$.</p>
<p>This would be easy <strong>if we knew the language of each tweet</strong>, since we could estimate</p>
<ul>
<li>$P(xyz| language = English)$ as #(number of times “xyz” is a trigram in the English tweets) / #(total trigrams in the English tweets)</li>
<li>$P(language = English)$ as the proportion of English tweets.</li>
</ul>
<p>Or, it would also be easy <strong>if we knew the n-gram probabilities for each language</strong>, since we could use Bayes’ theorem to compute the language <em>probabilities</em> for each tweet, and then take a weighted variant of the previous paragraph.</p>
<p><strong>The problem is that we know neither of these.</strong> So what the EM algorithm says is that that we can simply <strong>guess</strong>:</p>
<ul>
<li>Pretend we know the language of each tweet (by randomly assigning them at the beginning).</li>
<li>Using this guess, we can compute the n-gram probabilities for each language.</li>
<li>Using the n-gram probabilities for each language, we can recompute the language probabilities of each tweet.</li>
<li>Using these recomputed language probabilities, we can recompute the n-gram probabilities.</li>
<li>And so on, recomputing the language probabilities and n-gram probabilities over and over. While our guesses will be off in the beginning, the probabilities will eventually converge to (locally) minimize the likelihood. (In my tests, my language detector would sometimes correctly converge to an English detector, and sometimes it would converge to an English-and-Dutch detector.)</li>
</ul>
<h2>EM Analogy for the Layman</h2>
<p>Why does this work? Suppose you suddenly move to New York, and you want a way to differentiate between tourists and New Yorkers based on their activities. Initially, you don’t know who’s a tourist and who’s a New Yorker, and you don’t know which are touristy activities and which are not. So you randomly place people into two groups A and B. (You randomly assign all tweets to a language)</p>
<p>Now, given all the people in group A, you notice that a large number of them visit the Statue of Liberty; similarly, you notice that a large number of people in group B walk really quickly. (You notice that one set of words often has the n-gram “ing”, and that another set of words often has the n-gram “ias”; that is, you fix the language probabilities for each tweet, and recompute the n-gram probabilities for each language.)</p>
<p>So you start to put people visiting the Statue of Liberty in group A, and you start to put fast walkers in group B. (You fix the n-gram probabilities for each language, and recompute the language probabilities for each tweet.)</p>
<p>With your new A and B groups, you notice more differentiating factors: group A people tend to carry along cameras, and group B people tend to be more finance-savvy.</p>
<p>So you start to put camera-carrying folks in group A, and finance-savvy folks in group B.</p>
<p>And so on. Eventually, you settle on two groups of people and differentiating activities: people who walk slowly and visit the Statue of Liberty, and busy-looking people who walk fast and don’t visit. Assuming there are more native New Yorkers than tourists, you can then guess that the natives are the larger group.</p>
<h1>Results</h1>
<p>I wrote some Ruby code to implement the above algorithm, and trained it on half a million tweets, using English and “not English” as my two languages. The results looked surprisingly good from just eyeballing:</p>
<p><a href="https://img.skitch.com/20110303-qfrnb8gstgheh4xech4iutfskd.jpg"><img src="https://img.skitch.com/20110303-qfrnb8gstgheh4xech4iutfskd.jpg" alt="Example Results" /></a></p>
<p>But in order to get some hard metrics and to tune parameters (e.g., n-gram size), I needed a labeled dataset. So I pulled a set of English-language and Spanish-language documents from Project Gutenberg, and split them to form training and test sets (the training set consisted of 2000 lines of English and 1000 lines of Spanish, and 1000 lines of English and 1000 lines of Spanish for the test set).</p>
<p>Trained on bigrams, the detector resulted in:</p>
<ul>
<li>991 true positives (English lines correctly classified as English)</li>
<li>9 false negatives (English lines incorrectly classified as Spanish</li>
<li>11 false positives (Spanish lines incorrectly classified as English)</li>
<li>989 true negatives (Spanish lines correctly classified as English)</li>
</ul>
<p>for a precision of 0.989 and a recall of 0.991.</p>
<p>Trained on trigrams, the detector resulted in:</p>
<ul>
<li>992 true positives</li>
<li>8 false negatives</li>
<li>10 false positives</li>
<li>990 true negatives</li>
</ul>
<p>for a precision of 0.990 and a recall of 0.992.</p>
<p>Also, when I looked at the sentences the detector was making errors on, I saw that they almost always consisted of only one or two words (e.g., the incorrectly classified sentences were lines like “inmortal”, “autumn”, and “salir”). So the detector pretty much never made a mistake on a normal sentence!</p>
<h1>Code/Demo</h1>
<p>I put the code on <a href="https://github.com/echen/unsupervised-language-identification">my Github account</a>, and a quick <a href="http://babel-fett.heroku.com/">demo app</a>, trained on trigrams from tweets with lang=”en” according to the Twitter API, is <a href="http://babel-fett.heroku.com/">here</a>.</p>
</div>
Choosing a Machine Learning Classifier2011-04-27T04:15:00+02:002011-04-27T04:15:00+02:00Edwin Chentag:blog.echen.me,2011-04-27:/2011/04/27/choosing-a-machine-learning-classifier/
<div class="entry-content"><p>How do you know what machine learning algorithm to choose for your classification problem? Of course, if you really care about accuracy, your best bet is to test out a couple different ones (making sure to try different parameters within each algorithm as well), and select the best one by …</p></div>
<div class="entry-content"><p>How do you know what machine learning algorithm to choose for your classification problem? Of course, if you really care about accuracy, your best bet is to test out a couple different ones (making sure to try different parameters within each algorithm as well), and select the best one by cross-validation. But if you’re simply looking for a “good enough” algorithm for your problem, or a place to start, here are some general guidelines I’ve found to work well over the years.</p>
<h1>How large is your training set?</h1>
<p>If your training set is small, high bias/low variance classifiers (e.g., Naive Bayes) have an advantage over low bias/high variance classifiers (e.g., kNN), since the latter will overfit. But low bias/high variance classifiers start to win out as your training set grows (they have lower asymptotic error), since high bias classifiers aren’t powerful enough to provide accurate models.</p>
<p>You can also think of this as a generative model vs. discriminative model distinction.</p>
<h1>Advantages of some particular algorithms</h1>
<p><strong>Advantages of Naive Bayes:</strong> Super simple, you’re just doing a bunch of counts. If the NB conditional independence assumption actually holds, a Naive Bayes classifier will converge quicker than discriminative models like logistic regression, so you need less training data. And even if the NB assumption doesn’t hold, a NB classifier still often does a great job in practice. A good bet if want something fast and easy that performs pretty well. Its main disadvantage is that it can’t learn interactions between features (e.g., it can’t learn that although you love movies with Brad Pitt and Tom Cruise, you hate movies where they’re together).</p>
<p><strong>Advantages of Logistic Regression:</strong> Lots of ways to regularize your model, and you don’t have to worry as much about your features being correlated, like you do in Naive Bayes. You also have a nice probabilistic interpretation, unlike decision trees or SVMs, and you can easily update your model to take in new data (using an online gradient descent method), again unlike decision trees or SVMs. Use it if you want a probabilistic framework (e.g., to easily adjust classification thresholds, to say when you’re unsure, or to get confidence intervals) or if you expect to receive more training data in the future that you want to be able to quickly incorporate into your model.</p>
<p><strong>Advantages of Decision Trees:</strong> Easy to interpret and explain (for some people – I’m not sure I fall into this camp). They easily handle feature interactions and they’re non-parametric, so you don’t have to worry about outliers or whether the data is linearly separable (e.g., decision trees easily take care of cases where you have class A at the low end of some feature x, class B in the mid-range of feature x, and A again at the high end). One disadvantage is that they don’t support online learning, so you have to rebuild your tree when new examples come on. Another disadvantage is that they easily overfit, but that’s where ensemble methods like random forests (or boosted trees) come in. Plus, random forests are often the winner for lots of problems in classification (usually slightly ahead of SVMs, I believe), they’re fast and scalable, and you don’t have to worry about tuning a bunch of parameters like you do with SVMs, so they seem to be quite popular these days.</p>
<p><strong>Advantages of SVMs:</strong> High accuracy, nice theoretical guarantees regarding overfitting, and with an appropriate kernel they can work well even if you’re data isn’t linearly separable in the base feature space. Especially popular in text classification problems where very high-dimensional spaces are the norm. Memory-intensive, hard to interpret, and kind of annoying to run and tune, though, so I think random forests are starting to steal the crown.</p>
<h1>But…</h1>
<p>Recall, though, that better data often beats better algorithms, and designing good features goes a long way. And if you have a huge dataset, then whichever classification algorithm you use might not matter so much in terms of classification performance (so choose your algorithm based on speed or ease of use instead).</p>
<p>And to reiterate what I said above, if you really care about accuracy, you should definitely try a bunch of different classifiers and select the best one by cross-validation. Or, to take a lesson from the Netflix Prize (and Middle Earth), just use an ensemble method to choose them all.</p>
</div>
Kickstarter Data Analysis: Success and Pricing2011-04-25T04:15:00+02:002011-04-25T04:15:00+02:00Edwin Chentag:blog.echen.me,2011-04-25:/2011/04/25/kickstarter-data-analysis-success-and-pricing/
<div class="entry-content"><p><a href="http://www.kickstarter.com/">Kickstarter</a> is an online crowdfunding platform for launching creative projects. When starting a new project, project owners specify a deadline and the minimum amount of money they need to raise. They receive the money (less a transaction fee) only if they reach or exceed that minimum; otherwise, no money changes …</p></div>
<div class="entry-content"><p><a href="http://www.kickstarter.com/">Kickstarter</a> is an online crowdfunding platform for launching creative projects. When starting a new project, project owners specify a deadline and the minimum amount of money they need to raise. They receive the money (less a transaction fee) only if they reach or exceed that minimum; otherwise, no money changes hands.</p>
<p>What’s particularly fun about Kickstarter is that in contrast to <a href="http://www.kiva.org/">that other microfinance site</a>, Kickstarter projects don’t ask for loans; instead, patrons receive pre-specified rewards unique to each project. For example, someone donating money to help an artist record an album might receive a digital copy of the album if they donate 20 dollars, or a digital copy plus a signed physical cd if they donate 50 dollars.</p>
<p>There are <a href="http://www.kickstarter.com/discover/hall-of-fame?ref=sidebar">a bunch</a> of <a href="http://www.kickstarter.com/projects/1104350651/tiktok-lunatik-multi-touch-watch-kits">neat</a> <a href="http://www.kickstarter.com/projects/2024077040/neil-gaimans-the-price">projects</a>, and I’m tempted to put one of my own on there soon, so I thought it would be fun to gather some data from the site and see what makes a project successful.</p>
<h1>Categories</h1>
<p>I started by scraping the categories section.</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/kickstarter/successful-projects-by-category.png"><img src="http://dl.dropbox.com/u/10506/blog/kickstarter/successful-projects-by-category.png" alt="Successful projects by category" /></a></p>
<p>In true indie fashion, the artsy categories tend to dominate. (I’m surprised/disappointed how little love the Technology category gets.)</p>
<h1>Ending Soon</h1>
<p>The categories section really only provides a history of <em>successful</em> projects, though, so to get some data on unsuccessful projects as well, I took a look at the <a href="http://www.kickstarter.com/discover/ending-soon?ref=sidebar">Ending Soon</a> section of projects whose deadlines are about to pass.</p>
<p>It looks like about 50% of all Kickstarter projects get successfully funded by the deadline:</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/kickstarter/ending-soon-success.png"><img src="http://dl.dropbox.com/u/10506/blog/kickstarter/ending-soon-success.png" alt="Successful projects as deadline approaches" /></a></p>
<p>Interestingly, most of the final funding seems to happen in the final few days: with just 5 days left, only about 20% of all projects have been funded. (In other words, with just 5 days left, 60% of the projects that will eventually be successful are still unfunded.) So the approaching deadline seems to really spur people to donate. I wonder if it’s because of increased publicity in the final few days (the project owners begging everyone for help!) or if it’s simply procrastination in action (perhaps people want to wait to see if their donation is really necessary)?</p>
<p>Lesson: if you’re still not fully funded with only a couple days remaining, don’t despair.</p>
<h1>Success vs. Failure</h1>
<p>What factors lead a project to succeed? Are there any quantitative differences between projects that eventually get funded and those that don’t?</p>
<p>Two simple (if kind of obvious) things I noticed are that unsuccessful projects tend to require a larger amount of money:</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/kickstarter/successful-vs-unsuccessful-goal.png"><img src="http://dl.dropbox.com/u/10506/blog/kickstarter/successful-vs-unsuccessful-goal.png" alt="Unsuccessful projects tend to ask for more money" /></a></p>
<p>and unsuccessful projects also tend to raise less money in absolute terms (i.e., it’s not just that they ask for too much money to reach their goal – they’re simply not receiving enough money as well):</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/kickstarter/successful-vs-unsuccessful-amount-pledged.png"><img src="http://dl.dropbox.com/u/10506/blog/kickstarter/successful-vs-unsuccessful-amount-pledged.png" alt="Unsuccessful projects received less money" /></a></p>
<p>Not terribly surprising, but it’s good to confirm (and I’m still working on finding other predictors).</p>
<h1>Pledge Rewards</h1>
<p>There’s a lot of interesting work in behavioral economics on pricing and choice – for example, the <a href="http://youarenotsosmart.com/2010/07/27/anchoring-effect/">anchoring effect</a> suggests that when building a menu, you should <a href="http://www.neurosciencemarketing.com/blog/articles/neuro-menus-and-restaurant-psychology.htm">include an expensive item</a> to make other menu items look reasonably priced in comparison, and the <a href="http://en.wikipedia.org/wiki/The_Paradox_of_Choice:_Why_More_Is_Less">paradox of choice </a> suggests that too many choices lead to a decision freeze – so one aspect of the Kickstarter data I was especially interested in was how pricing of rewards affects donations. For example, does pricing the lowest reward at 25 dollars lead to more money donated (people don’t lowball at 5 dollars instead) or less money donated (25 dollars is more money than most people are willing to give)? And what happens if a new reward at 5 dollars is added – again, does it lead to more money (now people can donate something they can afford) or less money (the people that would have paid 25 dollars switch to a 5 dollar donation)?</p>
<p>First, here’s a look at the total number of pledges at each price. (More accurately, it’s the number of claimed rewards at each price.) [Update: the original version of this graph was wrong, but I’ve since fixed it.]</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/kickstarter/pledge%20amounts.png"><img src="http://dl.dropbox.com/u/10506/blog/kickstarter/pledge%20amounts.png" alt="Pledge Amounts" /></a></p>
<p>Surprisingly, 5 dollar and 1 dollar donations are actually not the most common contribution.</p>
<p>To investigate pricing effects, I started by looking at all (successful) projects that had a reward priced at 1 dollar, and compared the number of donations at 1 dollar with the number of donations at the next lowest reward.</p>
<p>Up to about 15-20 dollars, there’s a steady increase in the proportion of people who choose the second reward over the first reward, but after that, the proportion decreases.</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/kickstarter/anchoring.png"><img src="http://dl.dropbox.com/u/10506/blog/kickstarter/anchoring.png" alt="Anchoring" /></a></p>
<p><a href="http://dl.dropbox.com/u/10506/blog/kickstarter/anchoring-abline-b.png"><img src="http://dl.dropbox.com/u/10506/blog/kickstarter/anchoring-abline-b.png" alt="Anchoring with Regression Lines" /></a></p>
<p>So this perhaps suggests that if you’re going to price your lowest reward at 1 dollar, your next reward should cost roughly 20 dollars (or slightly more, to maximize your total revenue). Pricing above 20 dollars is a little too expensive for the folks who want to support you, but aren’t rich enough to throw gads of money; maybe rewards below 20 dollars aren’t good enough to merit the higher donation.</p>
<p>Next, I’m planning on digging a little deeper into pricing effects and what makes a project successful, so I’ll hopefully have some more Kickstarter analysis in a future post. In the meantime, in case anyone else wants to take a look, I put the data onto <a href="https://github.com/echen/kickstarter-data-analysis">my Github account</a>.</p>
</div>
A Mathematical Introduction to Least Angle Regression2011-04-21T04:15:00+02:002011-04-21T04:15:00+02:00Edwin Chentag:blog.echen.me,2011-04-21:/2011/04/21/a-mathematical-introduction-to-least-angle-regression/
<div class="entry-content"><p>(For a layman’s introduction, see <a href="http://blog.echen.me/2011/03/14/least-angle-regression-for-the-hungry-layman/">here</a>.)</p>
<p>Least Angle Regression (aka LARS) is a <strong>model selection method</strong> for linear regression (when you’re worried about overfitting or want your model to be easily interpretable). To motivate it, let’s consider some other model selection methods:</p>
<ul>
<li><strong>Forward selection</strong> starts with no …</li></ul></div>
<div class="entry-content"><p>(For a layman’s introduction, see <a href="http://blog.echen.me/2011/03/14/least-angle-regression-for-the-hungry-layman/">here</a>.)</p>
<p>Least Angle Regression (aka LARS) is a <strong>model selection method</strong> for linear regression (when you’re worried about overfitting or want your model to be easily interpretable). To motivate it, let’s consider some other model selection methods:</p>
<ul>
<li><strong>Forward selection</strong> starts with no variables in the model, and at each step it adds to the model the variable with the most explanatory power, stopping if the explanatory power falls below some threshold. This is a fast and simple method, but it can also be too greedy: we fully add variables at each step, so correlated predictors don’t get much of a chance to be included in the model. (For example, suppose we want to build a model for the deliciousness of a PB&J sandwich, and two of our variables are the amount of peanut butter and the amount of jelly. We’d like both variables to appear in our model, but since amount of peanut butter is (let’s assume) strongly correlated with the amount of jelly, once we fully add peanut butter to our model, jelly doesn’t add much explanatory power anymore, and so it’s unlikely to be added.)</li>
<li><strong>Forward stagewise regression</strong> tries to remedy the greediness of forward selection by only partially adding variables. Whereas forward selection finds the variable with the most explanatory power and goes all out in adding it to the model, forward stagewise finds the variable with the most explanatory power and updates its weight by only epsilon in the correct direction. (So we might first increase the weight of peanut butter a little bit, then increase the weight of peanut butter again, then increase the weight of jelly, then increase the weight of bread, and then increase the weight of peanut butter once more.) The problem now is that we have to make a ton of updates, so forward stagewise can be very inefficient.</li>
</ul>
<p>LARS, then, is essentially forward stagewise made fast. Instead of making tiny hops in the direction of one variable at a time, LARS makes optimally-sized leaps in optimal directions. These directions are chosen to make equal angles (equal correlations) with each of the variables currently in our model. (We like peanut butter best, so we start eating it first; as we eat more, we get a little sick of it, so jelly starts looking equally appetizing, and we start eating peanut butter and jelly simultaneously; later, we add bread to the mix, etc.)</p>
<p>In more detail, LARS works as follows:</p>
<ul>
<li>Assume for simplicity that we’ve standardized our explanatory variables to have zero mean and unit variance, and that our response variable also has zero mean.</li>
<li>Start with no variables in your model.</li>
<li>Find the variable $ x_1 $ most correlated with the residual. (Note that the variable most correlated with the residual is equivalently the one that makes the least angle with the residual, whence the name.)</li>
<li>Move in the direction of this variable until some other variable $ x_2 $ is just as correlated.</li>
<li>At this point, start moving in a direction such that the residual stays equally correlated with $ x_1 $ and $ x_2 $ (i.e., so that the residual makes equal angles with both variables), and keep moving until some variable $ x_3 $ becomes equally correlated with our residual.</li>
<li>And so on, stopping when we’ve decided our model is big enough.</li>
</ul>
<p>For example, consider the following image (slightly simplified from the <a href="http://www.stanford.edu/~hastie/Papers/LARS/LeastAngle_2002.pdf">original LARS paper</a>; $x_1, x_2$ are our variables, and $y$ is our response):</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/lars/lars-example.png"><img src="http://dl.dropbox.com/u/10506/blog/lars/lars-example.png" alt="LARS Example" /></a></p>
<p>Our model starts at $ \hat{\mu_0} $.</p>
<ul>
<li>The residual (the green line) makes a smaller angle with $ x_1 $ than with $ x_2 $, so we start moving in the direction of $ x_1 $.
At $ \hat{\mu_1} $, the residual now makes equal angles with $ x_1, x_2 $, and so we start moving in a new direction that preserves this equiangularity/equicorrelation.</li>
<li>If there were more variables, we’d change directions again once a new variable made equal angles with our residual, and so on.</li>
</ul>
<p>So when should you use LARS, as opposed to some other regularization method like lasso? There’s not really a clear-cut answer, but LARS tends to give very similar results as both lasso and forward stagewise (in fact, slight modifications to LARS give you lasso and forward stagewise), so I tend to just use lasso when I do these kinds of things, since the justifications for lasso make a little more sense to me. In fact, I don’t usually even think of LARS as a model selection method in its own right, but rather as a way to efficiently implement lasso (especially if you want to compute the full regularization path).</p>
</div>
Introduction to Cointegration and Pairs Trading2011-04-16T04:15:00+02:002011-04-16T04:15:00+02:00Edwin Chentag:blog.echen.me,2011-04-16:/2011/04/16/introduction-to-cointegration-and-pairs-trading/
<div class="entry-content"><h1>Introduction</h1>
<p>Suppose you see two drunks (i.e., two random walks) wandering around. The drunks don’t know each other (they’re independent), so there’s no meaningful relationship between their paths.</p>
<p>But suppose instead you have a drunk walking with her dog. This time there <em>is</em> a connection. What …</p></div>
<div class="entry-content"><h1>Introduction</h1>
<p>Suppose you see two drunks (i.e., two random walks) wandering around. The drunks don’t know each other (they’re independent), so there’s no meaningful relationship between their paths.</p>
<p>But suppose instead you have a drunk walking with her dog. This time there <em>is</em> a connection. What’s the nature of this connection? Notice that although each path individually is still an unpredictable random walk, given the location of one of the drunk or dog, we have a pretty good idea of where the other is; that is, the distance between the two is fairly predictable. (For example, if the dog wanders too far away from his owner, she’ll tend to move in his direction to avoid losing him, so the two stay close together despite a tendency to wander around on their own.) We describe this relationship by saying that the drunk and her dog form a cointegrating pair.</p>
<p>In more technical terms, if we have two non-stationary time series X and Y that become stationary when differenced (these are called integrated of order one series, or I(1) series; random walks are one example) such that some linear combination of X and Y is stationary (aka, I(0)), then we say that X and Y are cointegrated. In other words, while neither X nor Y alone hovers around a constant value, some combination of them does, so we can think of cointegration as describing a particular kind of long-run equilibrium relationship. (The definition of cointegration can be extended to multiple time series, with higher orders of integration.)</p>
<p>Other examples of cointegrated pairs:</p>
<ul>
<li>Income and consumption: as income increases/decreases, so too does consumption.</li>
<li>Size of police force and amount of criminal activity</li>
<li>A book and its movie adaptation: while the book and the movie may differ in small details, the overall plot will remain the same.</li>
<li>Number of patients entering or leaving a hospital</li>
</ul>
<h1>An application</h1>
<p>So why do we care about cointegration? In quantitative finance, cointegration forms the basis of the pairs trading strategy: suppose we have two cointegrated stocks X and Y, with the particular (for concreteness) cointegrating relationship X - 2Y = Z, where Z is a stationary series of zero mean. For example, X could be McDonald’s, Y could be Burger King, and the cointegration relationship would mean that X tends to be priced twice as high as Y, so that when X is more than twice the price of Y, we expect X to move down or Y to move up in the near future (and analogously, if X is less than twice the price of Y, we expect X to move up or Y to move down). This suggests the following trading strategy: if X - 2Y > d, for some positive threshold d, then we should sell X and buy Y (since we expect X to decrease in price and Y to increase), and similarly, if X - 2Y < -d, then we should buy X and sell Y.</p>
<h1>Spurious regression</h1>
<p>But why do we need the notion of cointegration at all? Why can’t we simply use, say, the R-squared between X or Y to see if X and Y have some kind of relationship? The reason is that standard regression analysis fails when dealing with non-stationary variables, leading to spurious regressions that suggest relationships even when there are none.</p>
<p>For example, suppose we regress two independent random walks against each other, and test for a linear relationship. A large percentage of the time, we’ll find high R-squared values and low p-values when using standard OLS statistics, even though there’s absolutely no relationship between the two random walks. As an illustration, here I simulated 1000 pairs of random walks of length 100, and found p-values less than 0.05 in 77% of the cases:</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/cointegration/spurious-regression.png"><img src="http://dl.dropbox.com/u/10506/blog/cointegration/spurious-regression.png" alt="Spurious Regression" /></a></p>
<h1>A Cointegration Test</h1>
<p>So how do you detect cointegration? There are several different methods, but the simplest is the Engle-Granger test, which works roughly as follows:</p>
<ul>
<li>Check that $ X_t $ and $ Y_t $ are both I(1).</li>
<li>Estimate the cointegrating relationship $ Y_t = aX_t + e_t $ by ordinary least squares.</li>
<li>Check that the cointegrating residuals $ e_t $ are stationary (say, by using a so-called unit root test, e.g., the Dickey-Fuller test).</li>
</ul>
<h1>Error-correction and Granger representation</h1>
<p>Something else that should perhaps be mentioned is the relationship between cointegration and error-correction mechanisms: suppose we have two cointegrated series $ X_t, Y_t $, with autoregressive representations</p>
<p>$ X_t = a X_{t-1} + b Y_{t-1} + u_t $
$ Y_t = c X_{t-1} + d Y_{t-1} + v_t $</p>
<p>By the Granger representation theorem (which is actually a bit more general than this), we then have</p>
<p>$ \Delta X_t = \alpha_1 (Y_{t-1} - \beta X_{t-1}) + u_t $
$ \Delta Y_t = \alpha_2 (Y_{t-1} - \beta X_{t-1}) + v_t $</p>
<p>where $ Y_{t-1} - \beta X_{t-1} \sim I(0) $ is the cointegrating relationship. Regarding $ Y_{t-1} - \beta X_{t-1} $ as the extent of disequilibrium from the long-run relationship, and the $ \alpha_i $ as the speed (and direction) at which the time series correct themselves from this disequilibrium, we can see that this formalizes the way cointegrated variables adjust to match their long-run equilbrium.</p>
<h1>Summary</h1>
<p>So, just to summarize a bit, cointegration is an equilibrium relationship between time series that individually aren’t in equilbrium (you can kind of contrast this with (Pearson) correlation, which describes a linear relationship), and it’s useful because it allows us to incorporate both short-term dynamics (deviations from equilibrium) and long-run expectations (corrections to equilibrium).</p>
</div>
Counting Clusters2011-03-14T04:15:00+01:002011-03-14T04:15:00+01:00Edwin Chentag:blog.echen.me,2011-03-14:/2011/03/14/counting-clusters/
<div class="entry-content"><p>Given a set of datapoints, we often want to know how many clusters the datapoints form. The <strong>gap statistic</strong> and the <strong>prediction strength</strong> are two practical algorithms for choosing the number of clusters.</p>
<h1>Gap Statistic</h1>
<p>The <a href="http://www.stanford.edu/~hastie/Papers/gap.pdf">gap statistic algorithm</a> works as follows:</p>
<p>For each i from 1 up to some …</p></div>
<div class="entry-content"><p>Given a set of datapoints, we often want to know how many clusters the datapoints form. The <strong>gap statistic</strong> and the <strong>prediction strength</strong> are two practical algorithms for choosing the number of clusters.</p>
<h1>Gap Statistic</h1>
<p>The <a href="http://www.stanford.edu/~hastie/Papers/gap.pdf">gap statistic algorithm</a> works as follows:</p>
<p>For each i from 1 up to some maximum number of clusters,</p>
<ol>
<li><p>Run a k-means algorithm on the original dataset to find i clusters, and sum the distance of all points from their cluster mean. Call this sum the <strong>dispersion</strong>.</p></li>
<li><p>Generate a set of <em>reference</em> datasets (of the same size as the original). One simple way of generating a reference dataset is to sample uniformly from the original dataset’s bounding rectangle; a more sophisticated approach is take into account the original dataset’s shape by sampling, say, from a rectangle formed from the original dataset’s principal components.</p></li>
<li><p>Calculate the dispersion of each of these reference datasets, and take their mean.</p></li>
<li><p>Define the ith <strong>gap</strong> by: log(mean dispersion of reference datasets) - log(dispersion of original dataset).</p></li>
</ol>
<p>Once we’ve calculated all the gaps (we can add confidence intervals as well; see <a href="http://www.stanford.edu/~hastie/Papers/gap.pdf">the original paper</a> for the formula), we can select the number of clusters to be the one that gives the maximum gap. (Sidenote: I view the gap statistic as a very statistical-minded algorithm, since it compares the original dataset against a set of reference “control” datasets.)</p>
<p>For example, here I’ve generated three Gaussian clusters:</p>
<p><a href="https://github.com/echen/gap-statistic/raw/master/examples/3_clusters_2d.png"><img src="https://github.com/echen/gap-statistic/raw/master/examples/3_clusters_2d.png" alt="Three Gaussian Clusters" /></a></p>
<p>And running the gap statistic algorithm, we see that it correctly detects the number of clusters to be three:</p>
<p><a href="https://github.com/echen/gap-statistic/raw/master/examples/3_clusters_2d_gaps.png"><img src="https://github.com/echen/gap-statistic/raw/master/examples/3_clusters_2d_gaps.png" alt="Gap Statistic on Three Gaussian Clusters" /></a></p>
<p>For a sample R implementation of the gap statistic, see the Github repository <a href="https://github.com/echen/gap-statistic">here</a>.</p>
<h1>Prediction Strength</h1>
<p>Another cluster-counting algorithm is the <a href="http://www-stat.stanford.edu/~tibs/ftp/predstr.ps">prediction strength algorithm</a>. In contrast to the gap statistic (which, as mentioned above, I find very statistically), I see prediction strength as taking a more machine learning viewpoint, since it’s formulated as a supervised learning problem validated against a test set.</p>
<p>To calculate prediction strength, for each i from 1 up to some maximum number of clusters:</p>
<ol>
<li><p>Divide the dataset into two groups, a training set and a test set.</p></li>
<li><p>Run a k-means algorithm on each set to find i clusters.</p></li>
<li><p>For each <em>test</em> cluster, count the proportion of pairs of points in that cluster that would remain in the same cluster, if each were assigned to its closest <em>training</em> cluster mean.</p></li>
<li><p>The minimum over these proportions is the <strong>prediction strength</strong> for i clusters.</p></li>
</ol>
<p>Once we’ve calculated the prediction strength for each number of clusters, we select the number of clusters to be the maximum i such that the prediction strength for i is greater than some threshold. (The paper suggests 0.8 - 0.9 as a good threshold, and I’ve seen 0.8 work well in practice.)</p>
<p>Here’s the prediction strength algorithm run on the same example above:</p>
<p><a href="https://github.com/echen/prediction-strength/raw/master/examples/3_clusters_2d_ps.png"><img src="https://github.com/echen/prediction-strength/raw/master/examples/3_clusters_2d_ps.png" alt="Prediction Strength on Three Gaussian Clusters" /></a></p>
<p>Again, check out a sample R implementation of the prediction strength <a href="https://github.com/echen/prediction-strength">here</a>.</p>
<p>In practice, I tend to prefer using the gap statistic algorithm, since it’s a little easier to code and it doesn’t require selecting an arbitrary threshold like the prediction strength does. I’ve also found that it gives slightly better results (though the original prediction strength paper has the opposite finding).</p>
<h1>Appendix</h1>
<p>I ended up giving a brief description of two very common clustering algorithms, <strong>k-means</strong> and <strong>Gaussian mixture models</strong> in the comments, so I figured I might as well bring them up here.</p>
<h2>k-means algorithm</h2>
<p>Suppose we have a set of datapoints that we want to cluster. We want to learn two things:</p>
<ul>
<li>A description of the clusters themselves (so that if new points come in, we can assign them to a cluster).</li>
<li>Which clusters our current points fall into.</li>
</ul>
<p>We start by initializing k cluster centers (e.g., by randomly choosing k points among our datapoints). Then we repeatedly</p>
<ul>
<li><strong>Step A</strong>: Assign each datapoint to the nearest cluster center.</li>
<li><strong>Step B</strong>: Update all the cluster centers: for each cluster i, take the mean over all points currently in the cluster, and update cluster center i to be this mean.</li>
<li>(Repeat steps A and B above until the cluster assignments stop changing.)</li>
</ul>
<p>And that’s pretty much it for k-means.</p>
<h2>k-means from an EM point of View</h2>
<p>To ease the transition into Gaussian mixture models,
let’s also describe the k-means algorithm using <a href="http://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm">EM</a> language.</p>
<p>Note that if we knew for certain either 1) the exact cluster centers or 2) the cluster each point belonged to, we could trivially solve k-means, since</p>
<ul>
<li>If we knew the exact cluster centers, all we’d have to do is assign each point to its nearest cluster center, and we’d be done.</li>
<li>If we knew which cluster each point belonged to, we could pick the cluster center by simply taking the mean over all points in that cluster.</li>
</ul>
<p>The problem is that we know <em>neither</em> of these, and so we alternate between making educated guesses of each one:</p>
<ul>
<li>In A step above, we pretend that we know the cluster centers, and based off this pretense, we guess which cluster each point belongs to. (This is also known as the <strong>E step</strong> in the EM algorithm.)</li>
<li>In the B step above, we do the reverse: we pretend that we know which cluster each point belongs to, and then try to guess the cluster centers. (This is also known as the <strong>M step</strong> in EM.)</li>
</ul>
<p>Our guesses keep getting better and better, and eventually we’ll converge.</p>
<h2>Gaussian Mixture Models</h2>
<p>k-means has a <em>hard</em> notion of clustering: point X either belongs to cluster C or it doesn’t. But sometimes we want a <em>soft</em> notion instead: point X belongs to cluster C with probability p (according to a Gaussian kernel). This is where <a href="http://en.wikipedia.org/wiki/Mixture_model">Gaussian mixture modeling</a> comes in.</p>
<p>To run a GMM, we start by initializing $k$ Gaussians (say, by randomly choosing $k$ points to be the centers of the Gaussians and by setting the variance of each Gaussians to be the overall variance), and then we repeatedly:</p>
<ul>
<li><strong>E Step</strong>: Pretend we know the parameters of each Gaussian cluster, and assign each datapoint to Gaussian cluster i with appropriate probability.</li>
<li><strong>M Step</strong>: Pretend we know the probabilities that each point belongs to a given cluster. Using these probabilities, update the means and variances of each Gaussian cluster: the new mean for cluster i is the weighted mean over all points (where the weight of each point X is the probability that X belongs to cluster i), and similarly for the new variance.</li>
</ul>
<p>This is exactly like k-means in the EM formulation, except we replace the binary clustering formula with Gaussian kernels.</p>
</div>
Hacker News Analysis2011-03-14T04:15:00+01:002011-03-14T04:15:00+01:00Edwin Chentag:blog.echen.me,2011-03-14:/2011/03/14/hacker-news-analysis/
<div class="entry-content"><p>I was playing around with the <a href="http://api.ihackernews.com/?db">Hacker News database</a> <a href="http://ronnieroller.com/">Ronnie Roller</a> made (thanks!), so I thought I’d post some of the things I looked at.</p>
<h1>Activity on the Site</h1>
<p>My first question was how activity on the site has increased over time. I looked at number of posts, points …</p></div>
<div class="entry-content"><p>I was playing around with the <a href="http://api.ihackernews.com/?db">Hacker News database</a> <a href="http://ronnieroller.com/">Ronnie Roller</a> made (thanks!), so I thought I’d post some of the things I looked at.</p>
<h1>Activity on the Site</h1>
<p>My first question was how activity on the site has increased over time. I looked at number of posts, points on posts, comments on posts, and number of users.</p>
<h2>Posts</h2>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/posts_by_month.png" alt="Hacker News Posts by Month" /></p>
<p>This looks like a strong linear fit, with an increase of 292 posts every month.</p>
<h2>Comments</h2>
<p>For comments, I fit a quadratic regression:</p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/comments_by_month.png" alt="Hacker News Comments by Month" /></p>
<h2>Points</h2>
<p>A quadratic regression was also a better fit for points by month:</p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/points_by_month.png" alt="Hacker News Points by Month" /></p>
<h2>Users</h2>
<p>And again for the number of distinct users with a submission:</p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/users.png" alt="Hacker News Users by Month" /></p>
<h1>Points and Comments</h1>
<p>My next question was how points and comments related. Intuitively, posts with more points should have more comments, but it’s nice to check (maybe really good posts are kind of boring, so don’t lead to much discussion).</p>
<p>First, I plotted the points and comments of each individual post:</p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/all_points_vs_comments.png" alt="All Points vs. Comments" /></p>
<p>As expected, there’s an overall positive correlation between points and comments. Interestingly, there are quite a few high-points posts with no comments.</p>
<p>The plot’s quite noisy, though, so let’s try cleaning it up a bit, by taking the median number of comments per points level (and removing posts at the higher end, where we have little data):</p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/points_vs_median_comments.png" alt="Points vs. Median Comments" /></p>
<p>We see that posts with more points do tend to have more comments. Also, variance in number of comments is indicated by size and color, so (unsurprisingly) posts with more points have larger variance in their number of comments.</p>
<h1>Quality of Posts</h1>
<p>Another question was whether the quality of posts has degraded over time.</p>
<p>First, I computed a normalized “score” for each post, where a post’s score is defined as the number of points divided by the number of distinct users who made a submission in the same month. (The denominator is a rough proxy for the number of active users, and the goal of the score is to provide a way to compare posts across time.)</p>
<p>While the median score has declined over time (as perhaps should be expected, since only a fixed number of items can reach the front page):</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/hn-analysis/median-score.png"><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/median-score.png" alt="Median Score" /></a></p>
<p>the absolute <em>number</em> of quality posts, defined as posts with a score greater than the (admittedly arbitrarily chosen) threshold 0.01, has increased (until possibly a dip starting in 2010):</p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/num_quality_posts2.png" alt="Number of Quality Posts" /></p>
<p>(Of course, without some further analysis, it’s not clear how well this score measures quality of posts, so take these numbers with a grain of salt.)</p>
<h1>Company Trends</h1>
<p>Also, I wanted to see how certain topics have trended over time, so I looked at how mentions of some of the big-name companies (Google, Facebook, Microsoft, Yahoo, Twitter, Apple) have changed. For each company, I plotted the percentage of posts with the company’s name in the title, and also made a smoothed plot comparing all six at the end. Note that Microsoft and Yahoo seem to be trending slightly downward, and Apple seems to be trending upward.</p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/pct_microsoft.png" alt="Mentions of Microsoft" /></p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/pct_yahoo.png" alt="Mentions of Yahoo" /></p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/pct_google.png" alt="Mentions of Google" /></p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/pct_facebook.png" alt="Mentions of Facebook" /></p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/pct_twitter.png" alt="Mentions of Twitter" /></p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/pct_apple.png" alt="Mentions of Apple" /></p>
<p><img src="http://dl.dropbox.com/u/10506/blog/hn-analysis/all_trends2.png" alt="All Trends" /></p>
</div>
Layman's Introduction to Measure Theory2011-03-14T04:15:00+01:002011-03-14T04:15:00+01:00Edwin Chentag:blog.echen.me,2011-03-14:/2011/03/14/laymans-introduction-to-measure-theory/
<div class="entry-content"><p>Measure theory studies ways of generalizing the notions of length/area/volume. Even in 2 dimensions, it might not be clear how to measure the area of the following fairly tame shape:</p>
<p><img src="http://d2o7bfz2il9cb7.cloudfront.net/main-qimg-809c3bdb18539dfa2917ee766a0a6159" alt="What's the area of this shape?" /></p>
<p>much less the “area” of even weirder shapes in higher dimensions or different spaces entirely.</p>
<p>For example, suppose …</p></div>
<div class="entry-content"><p>Measure theory studies ways of generalizing the notions of length/area/volume. Even in 2 dimensions, it might not be clear how to measure the area of the following fairly tame shape:</p>
<p><img src="http://d2o7bfz2il9cb7.cloudfront.net/main-qimg-809c3bdb18539dfa2917ee766a0a6159" alt="What's the area of this shape?" /></p>
<p>much less the “area” of even weirder shapes in higher dimensions or different spaces entirely.</p>
<p>For example, suppose you want to measure the length of a book (so that you can get a good sense of how long it takes to read). What’s a good measure? One possibility is to measure a book’s length in <em>pages</em>. Since books provide page counts, this is a fairly easy measure to get. However, different versions of the same book (e.g., hardcover and paperback versions) tend to have different page counts, so this page measure doesn’t satisfy the nice property of version invariance (which we would like to have, since hardcover and paperback versions of the same book take the same time to read). Also, not all books even have page counts (think Kindle books), so this measure doesn’t allow us to measure the length of all books we might want to read.</p>
<p>Another, possibly better measure is to measure a book’s length in terms of the number of <em>words</em> it contains. Now we do have version invariance (hardcover and paperback versions contain the same number of words) and we can measure the length of Kindle books as well. We can even do things like add two books together, and the measure/number of words of the concatenated books will pleasantly equal the sum of the measures/number of words of each book alone.</p>
<p>However, what happens when we try to measure a picture book’s length in words? We can’t – picture books are too pathological. Maybe we could say that a picture book has measure zero (since a picture book has no words), but then we get unhappy things like books of measure zero taking a really long time to read (imagine a really long picture book). So maybe a better option is to say that picture books are simply unmeasurable. Whenever someone asks for the length of a picture book, we ignore them, and this way our measure will continue to be a good approximation of reading time and we get to keep our other nice properties as well.</p>
<p>Similarly, measure theory asks questions like:</p>
<ul>
<li>How do we define a measure on our space? (Jordan measure and Lebesgue measure are two different options in Euclidean space.)</li>
<li>What properties does our measure satisfy? (For example, does it satisfy translational invariance, rotational invariance, additivity?)</li>
<li>Which objects are measurable/which objects can we say it’s okay not to measure in order to preserve nice properties of our measure? (The Banach-Tarski ball can be rigidly reassembled into two copies of the same shape and size as the original, so we don’t want it to be measurable, since then we would lose additivity properties.)</li>
</ul>
<p>And once we’ve defined a “generalized area” (our measure), we can try to generalize other mathematical concepts as well. For example, recall that the (Riemann) integral that you learn in calculus measures the area under a curve. What happens if we replace the “area” in the Riemann integral with our new, generalized measure (e.g., to get the Lebesgue integral)? Measure theory also helps make certain probability statements mathematically precise (e.g., we can say exactly what it means that a fair coin flipped infinitely often will “almost never” land heads more than 50% of the time).</p>
</div>
Layman's Introduction to Random Forests2011-03-14T04:15:00+01:002011-03-14T04:15:00+01:00Edwin Chentag:blog.echen.me,2011-03-14:/2011/03/14/laymans-introduction-to-random-forests/
<div class="entry-content"><p>Suppose you’re very indecisive, so whenever you want to watch a movie, you ask your friend Willow if she thinks you’ll like it. In order to answer, Willow first needs to figure out what movies you like, so you give her a bunch of movies and tell her …</p></div>
<div class="entry-content"><p>Suppose you’re very indecisive, so whenever you want to watch a movie, you ask your friend Willow if she thinks you’ll like it. In order to answer, Willow first needs to figure out what movies you like, so you give her a bunch of movies and tell her whether you liked each one or not (i.e., you give her a labeled training set). Then, when you ask her if she thinks you’ll like movie X or not, she plays a 20 questions-like game with IMDB, asking questions like “Is X a romantic movie?”, “Does Johnny Depp star in X?”, and so on. She asks more informative questions first (i.e., she maximizes the information gain of each question), and gives you a yes/no answer at the end.</p>
<p>Thus, <strong>Willow is a decision tree for your movie preferences</strong>.</p>
<p>But Willow is only human, so she doesn’t always generalize your preferences very well (i.e., she overfits). In order to get more accurate recommendations, you’d like to ask a bunch of your friends, and watch movie X if most of them say they think you’ll like it. That is, instead of asking only Willow, you want to ask Woody, Apple, and Cartman as well, and they vote on whether you’ll like a movie (i.e., <strong>you build an ensemble classifier</strong>, aka a forest in this case).</p>
<p>Now you don’t want each of your friends to do the same thing and give you the same answer, so you first give each of them slightly different data. After all, you’re not absolutely sure of your preferences yourself – you told Willow you loved Titanic, but maybe you were just happy that day because it was your birthday, so maybe some of your friends shouldn’t use the fact that you liked Titanic in making their recommendations. Or maybe you told her you loved Cinderella, but actually you <em>really really</em> loved it, so some of your friends should give Cinderella more weight. So instead of giving your friends the same data you gave Willow, you give them slightly perturbed versions. You don’t change your love/hate decisions, you just say you love/hate some movies a little more or less (formally, <strong>you give each of your friends a bootstrapped version of your original training data</strong>). For example, whereas you told Willow that you liked Black Swan and Harry Potter and disliked Avatar, you tell Woody that you liked Black Swan so much you watched it twice, you disliked Avatar, and don’t mention Harry Potter at all.</p>
<p>By using this ensemble, you hope that while each of your friends gives somewhat idiosyncratic recommendations (Willow thinks you like vampire movies more than you do, Woody thinks you like Pixar movies, and Cartman thinks you just hate everything), the errors get canceled out in the majority. Thus, <strong>your friends now form a bagged (bootstrap aggregated) forest of your movie preferences</strong>.</p>
<p>There’s still one problem with your data, however. While you loved both Titanic and Inception, it wasn’t because you like movies that star Leonardio DiCaprio. Maybe you liked both movies for other reasons. Thus, you don’t want your friends to all base their recommendations on whether Leo is in a movie or not. So when each friend asks IMDB a question, only a random subset of the possible questions is allowed (i.e., <strong>when you’re building a decision tree, at each node you use some randomness in selecting the attribute to split on</strong>, say by randomly selecting an attribute or by selecting an attribute from a random subset). This means your friends aren’t allowed to ask whether Leonardo DiCaprio is in the movie whenever they want. So whereas previously you injected randomness at the data level, by perturbing your movie preferences slightly, now you’re injecting randomness at the model level, by making your friends ask different questions at different times.</p>
<p>And so <strong>your friends now form a random forest</strong>.</p>
</div>
Netflix Prize Summary: Factorization Meets the Neighborhood2011-03-14T04:15:00+01:002011-03-14T04:15:00+01:00Edwin Chentag:blog.echen.me,2011-03-14:/2011/03/14/netflix-prize-summary-factorization-meets-the-neighborhood/
<div class="entry-content"><p>(Way back when, I went through all the Netflix prize papers. I’m now (very slowly) trying to clean up my notes and put them online. Eventually, I hope to have a more integrated tutorial, but here’s a rough draft for now.)</p>
<p>This is a summary of Koren’s …</p></div>
<div class="entry-content"><p>(Way back when, I went through all the Netflix prize papers. I’m now (very slowly) trying to clean up my notes and put them online. Eventually, I hope to have a more integrated tutorial, but here’s a rough draft for now.)</p>
<p>This is a summary of Koren’s 2008 <a href="public.research.att.com/~volinsky/netflix/kdd08koren.pdf">Factorization Meets the Neighborhood: a Multifaceted Collaborative Filtering Model</a>.</p>
<p>There are two approaches to collaborative filtering: neighborhood methods and latent factor models.</p>
<ul>
<li>Neighborhood models are most effective at detecting very localized relationships (e.g., that people who like X-Men also like Spiderman), but poor at detecting a user’s overall signals.</li>
<li>Latent factor models are best at estimating overall structure (e.g., that a user likes horror movies), but are poor at detecting strong associations among small sets of closely related items.</li>
</ul>
<p>Since the two approaches have complementary strengths and weaknesses, we should integrate the two; this integration is the focus of this paper.</p>
<h1>Preliminaries</h1>
<p>As mentioned in previous papers, we should normalize out common effects from movies. Throughout the rest of this paper, Koren uses a baseline estimate of overall rating mean + user deviation from average + movie deviation from average for the rating of user i on movie i; estimation of the latter two parameters are done by solving a regularized least squares problem.</p>
<p>Koren then describes using a binary matrix (1 for rated, 0 for not rated) as a source of implicit feedback. This is useful because the mere fact that a user rated many science fiction movies (say) suggests that the user likes science fiction movies.</p>
<h1>A Neighborhood Model</h1>
<p>Recall the previous paper, where we modeled each rating $r_{ui}$ as</p>
<p>$$r_{ui} = b_{ui}+ \sum_{N \in N(i; u)} (r_{uj} - b_{uj}) w_{ij},$$</p>
<p>where $N(i; u)$ is the k items most similar to i among the items user u rated, and the $w_{ij}$ are parameters to be learned by solving a regularized least squares problem.</p>
<p>This paper makes several enhancements to that model. First, we replace $N(i; u)$ with $R^k(i; u)$, the intersection of the k items most similar to i (among all items) intersected with the items user u rated. Also, we denote by $N^k(i; u)$ the intersection of the k items most similar to i with the items user u has provided implicit feedback for. This gives us</p>
<p>$$r_{ui} = b_{ui} + \sum_{j \in R^k(i; u)} (r_{uj} - b_{uj}) w_{ij} + \sum_{j \in N^k(i; u)} c_{ij},$$</p>
<p>where the $c_{ij}$ are another set of parameters to learn.</p>
<p>Notice that by taking the intersection of the k items most similar to i with the items user u rated (giving perhaps a set of size less than k), rather than taking the k items most similar to i among the items user u rated, we let our model be influenced not only by what a user rates, but also by what a user does not rate. For example, if a user does not rate LOTR 1 or LOTR 2, his predicted rating for LOTR 3 is penalized.</p>
<p>This implies that our current model encourages greater deviations from baseline estimates for users that provided many ratings or plenty of implicit feedback. In other words, for well-modeled users with a lot of input, we are willing to predict quirkier and less common recommendations; users we have less information about, on the other hand, receive safer, baseline estimates.</p>
<p>Nonetheless, this dichotomy between power users and newbie users is perhaps overemphasized by our current model, so we moderate the dichotomy by modifying our model to be</p>
<p>$$r_{ui} = b_{ui} + |R^k(i; u)|^{-0.5} \sum_{j \in R^k(i; u)} (r_{uj} - b_{uj}) w_{ij} + |N^k(i; u)|^{-0.5} \sum_{j \in N^k(i; u)} c_{ij}.$$</p>
<p>Parameters are determined by solving a regularized least squares problem.</p>
<h1>Latent Factor Models Revisited</h1>
<p>Typical SVD approaches are based on the following rule:</p>
<p>$$r_{ui} = b_{ui} + p_u^T q_i,$$</p>
<p>where $p_u$ is a user-factors vector and $q_i$ is an item-factors vector. We describe two enhancements.</p>
<h2>Asymmetric-SVD</h2>
<p>One suggestion is to replace $p_u$ with</p>
<p>$$|R(u)|^{-0.5} + \sum_{j \in R(u)} (r_{uj} - b_{uj}) x_j + |N(u)|^{-0.5} \sum_{j \in N(u)} y_j,$$</p>
<p>where $R(u)$ is the set of items user u has rated, and $N(u)$ is the set of items user u has provided implicit feedback for. In other words, this model represents users through the items they prefer, rather than expressing users in a latent feature space. This model has several advantages:</p>
<ul>
<li>Asymmetric-SVD does not parameterize users, so we do not need to wait to retrain the model when a user comes in. Instead, we can handle new users as soon as they provide feedback.</li>
<li>Predictions are a direct function of past feedback, so we can easily explain predictions. (When using a pure latent feature solution, however, explainability is difficult.)</li>
</ul>
<p>As usual, parameters are learned via a regularized least-squares minimization.</p>
<h2>SVD++</h2>
<p>Another approach is to continue modeling users as latent features, while adding implicit feedback. Thus, we replace $p_u$ with $p_u + |N(u)|^{-0.5} \sum_{j \in N(u)} y_j$. While we lose the easily explainability and immediate feedback of the Asymmetric-SVD model, this approach is likely more accurate.</p>
<h1>An Integrated Model</h1>
<p>An integrated model incorporating baseline estimates, the neighborhood approach, and the latent factor approach is as follows:</p>
<p>$$r_{ui} = \left[\mu + b_u + b_i\right] +\left[q_i^T \big(p_u + \sqrt{|N(u)|}\sum_{j \in N(u)} y_j \big)\right] + \left[\sqrt{|R^k(i;u)} \sum_{j \in R^k(i; u)}(r_{uj} - b_{uj})w_{ij}+\sqrt{|N^k(i;u)|} \sum_{j \in N^k(i; u)} c_{ij}\right].$$</p>
<p>Note that we have used $(\mu + b_u + b_i)$ as our baseline estimate. We also used the SVD++ model, but we could use the Asymmetric-SVD model instead.</p>
<p>This rule provides a 3-tier model for recommendations:</p>
<ul>
<li>The first baseline group describes general properties of the item and user. For example, it may say that “The Sixth Sense” movie is known to be a good movie in general, and that Joe rates like the average user.</li>
<li>The next latent factor group may say that since “The Sixth Sense” and Joe rate high on the Psychological Thrillers Scale, Joe may like The Sixth Sense because he likes this genre of movies in general.</li>
<li>The final neighborhood tier makes fine-grained adjustments that are hard to file, such as the fact that Joe rated low the movie “Signs”, a similar psychological thriller by the same director.</li>
</ul>
<p>As usual, model parameters are determined by minimizing the regularized squared error function through gradient descent.</p>
</div>
Netflix Prize Summary: Scalable Collaborative Filtering with Jointly Derived Neighborhood Interpolation Weights2011-03-14T04:15:00+01:002011-03-14T04:15:00+01:00Edwin Chentag:blog.echen.me,2011-03-14:/2011/03/14/netflix-prize-summary-scalable-collaborative-filtering-with-jointly-derived-neighborhood-interpolation-weights/
<div class="entry-content"><p>(Way back when, I went through all the Netflix prize papers. I’m now (very slowly) trying to clean up my notes and put them online. Eventually, I hope to have a more integrated tutorial, but here’s a rough draft for now.)</p>
<p>This is a summary of Bell and …</p></div>
<div class="entry-content"><p>(Way back when, I went through all the Netflix prize papers. I’m now (very slowly) trying to clean up my notes and put them online. Eventually, I hope to have a more integrated tutorial, but here’s a rough draft for now.)</p>
<p>This is a summary of Bell and Koren’s 2007 <a href="public.research.att.com/~volinsky/netflix/BellKorICDM07.pdf">Scalable Collaborative Filtering with Jointly Derived Neighborhood Interpolation Weights</a> paper.</p>
<p><strong>tl;dr</strong> This paper’s main innovation is deriving neighborhood weights by solving a least squares problem, instead of using a standard similarity function to compute weights.</p>
<p>This paper improves upon the standard neighborhood approach to collaborative filtering in three areas: better data normalization, better neighbor weights (this is the key section), and better use of user data. I’ll first review the standard neighborhood approach, and follow with a description of these enhancements.</p>
<h2>Background: Standard Neighborhood Approach to Collaborative Filtering</h2>
<p>Recall that there are two types of neighborhood approaches:</p>
<ul>
<li>User-based approaches: to predict user i’s rating of item j, take the users most similar to user i, and perform a weighted average of their ratings of item j.</li>
<li>Item-based approaches: to predict user i’s rating of item j, perform a weighted average of user i’s ratings of items similar to item j.</li>
</ul>
<p>For example, to predict how you would rate the first Harry Potter movie, the user-based approach looks at how your friends rated the first Harry Potter movie, while the item-based approach looks at how you rated movies like Lord of the Rings and Twilight.</p>
<h2>Better Data Normalization</h2>
<p>Suppose I ask my friend Chris whether I should watch the latest Twilight movie. He tells me he would rate it 4.0/5 stars. Great, that’s a high rating, so that means I should watch it – or does it? It turns out that Chris is a super cheerful guy who’s never met a movie he didn’t like, and his average rating for a movie is actually 4.5/5 stars. So Twilight is actually less than average for him, and hence 4.0/5 stars from Chris isn’t actually that hearty a recommendation.</p>
<p>As another example, suppose you look at doctor ratings on Yelp. They’re abnormally high: the average is far from 3/5 stars. Why is this? Maybe it’s harder for people to change doctors than it is to go to a new restaurant, so people might not want to rate a doctor poorly when they know they’ll have to see the doctor again. Thus, an average rating of 5 stars on a McDonalds restaurant is much more impressive than an average of 5 stars on Dr. Joe.</p>
<p>The lesson is that when using existing ratings, we should normalize out these types of effects, so that ratings are as comparable as possible.</p>
<p>Another way of thinking about this is that we are simply building a regression model. That is, for each user u, we have a model
$r_{ui} = (\sum \theta_u x_{ui}) + SpecificRating$, where the $x_{ui}$ are common explanatory variables and we want to estimate $\theta_u$; and similarly for each item i. Once we’ve estimated the $\theta_u$, we can use the fancier neighborhood models on the specific ratings.</p>
<p>For example, suppose we want to predict Bob’s rating of Titanic. We’ve built a regression model with two explanatory variables, whether the movie was Oscar-nominated (1 if so, -1 if not) and whether the movie contains Kate Winslet (1 if so, -1 if not), and we’ve determined that Bob’s weights on these two variables are -2 (Bob tends to hate Oscar movies) and +1.5 (Bob likes Kate Winslet). Similarly, his friend John has weights +1 and -0.5 for these two variables (John likes Oscars, but dislikes Kate Winslet). So if we know that John rated Titanic a 4, then we have 4 = 1(1) + -0.5(1) + (John’s specific rating), so John’s specific rating of Titanic is 3.5. If we use John’s rating alone to estimate Bob’s, we might guess that Bob would rate Titanic -2(1) + 1.5(1) + (John’s specific rating) = 3.0.</p>
<p>To estimate the $\theta_u$, we actually perform this estimation in sequence: each explanatory variable is used to model the <em>residual</em> from the previous explanatory variable. Also, instead of using the maximum-likelihood unbiased estimator $\hat{\theta_u} = \frac{\sum r_{ui} x_{ui}}{x _ {ui} ^ 2}$, we shrink the weights to prevent overfitting. From a Bayesian point of view, the shrinkage arises from a hierarchical model where the true $\theta_u \sim N(\mu, \sigma^2)$, and $\hat{\theta_u} | \theta_u \sim N(\theta_u, \sigma_u^2)$, leading to $E(\theta_u | \hat{\theta_u}) = \frac{\sigma^2 \hat{\theta_u} + \sigma_u^2 \mu}{\sigma^2 + \sigma_u^2}$.</p>
<p>In practice, the explanatory variables Bell and Koren found to work well included the overall mean of all ratings, each movie’s specific mean, each user’s specific mean, time since movie release, time since user join, and number of ratings for each movie.</p>
<h2>Better Neighbor Weights</h2>
<p>Let’s consider some deficiencies of the neighborhood approach:</p>
<ul>
<li>Suppose I want to use the first LOTR movie to predict ratings of the first Harry Potter movie. To do this, I need to say how much weight the first LOTR movie should have in this prediction. But how do I choose this weight? Standard neighborhood approaches essentially pick arbitrary similarity functions (e.g., Pearson correlation, cosine distance) as the weight, possibly testing several similarity functions to see which gives the best performance, but is there a more principled approach to choosing weights?</li>
<li>The standard neighborhood approach ignores the fact that neighbors aren’t independent. For example, suppose all three LOTR movies are neighbors of the first HP movie. Since the three LOTR movies are so similar to each other, the standard approach is overcounting their information. Here’s an analogy: suppose I ask five of my friends where I should eat tonight. Three of them live together (boyfriend, girlfriend, and roommate), and they all recently took a trip together to Japan and are sick of Japanese food, so they vehemently recommend against sushi. Thus, my friends’ recommendations have a stronger bias than would appear if I asked five friends who didn’t know each other at all.</li>
</ul>
<p>We’ll see how using an optimization method to derive weights (as opposed to deriving weights via a similarity function) overcomes these two limitations.</p>
<p>Recall our problem: we want to predict $r_{ui}$, user u’s rating of item i, and what we have is a set $N(i; u)$ of K neighbors of item i that user u has also rated. (These K neighbors are selected via a similarity function, as is standard.) So what we want to do is find weights $w_{ij}$ such that $r_{ui} = \sum_{j \in N(i; u) w_{ij} r_{uj}}$. A natural approach, then, is simply to choose our weights to minimize $\min_w \sum_{v \neq u} \left( r_{vi} - \sum_{j \in N(i; u)} w_{ij} r_{vj}\right)^2$.</p>
<p>Notice how this optimization solves our two problems above: it’s not only a more principled approach (we choose our weights by minimizing squared error), but by deriving weights simultaneously, we overcome interaction effects.</p>
<p>Differentiating our cost function, we find that the optimal weights satisfy the equation $Aw = b$, where A is a $K \times K$ matrix defined by $A_{jk} = \sum_{v \neq u} r_{vj} r_{vk}$ and $b$ is a vector defined by $b_j = \sum_{v \neq u} r_{vj} r_{vi}$.</p>
<p>However, not all users have rated every movie, so some of the ratings may be missing from the above formulas. So we should instead use an estimate of A and b, such as $\bar{A}_{jk} = \frac{\sum_{v \in U(j,k)} r_{vj} r_{vk}}{|U(j, k)|}$, where $U(j, k)$ is the set of users who rated both j and k, and similarly for b. To avoid overfitting, we should further modify by shrinking to a common mean: $\hat{A}_{jk} = \frac{|U(J,K)|\bar{A}_{jk} + \beta A_{\mu}}{|U(j,k)| + \beta}$, where $\beta$ is a shrinkage parameter and $A_{\mu}$ is the mean over all $\bar{A}$, and similarly for b.</p>
<p>Note that another benefit of our optimization-derived weights is that the weights of neighbors are no longer constrained to sum to 1. Thus, if an item simply has no strong neighbors, the neighbors’ prediction will have only a small effect.</p>
<p>Also, when engineering these methods in practice, we should precompute all item-item similarities and all entries in the matrix $A$.</p>
<h2>Better Use of User Data</h2>
<p>Neighborhood models typically follow the item-based approach for two reasons:</p>
<ul>
<li>There are typically many more users than items, and new users come in much more frequently than new items, so it is easier to compute all pairs of item-item similarities.</li>
<li>Users have diverse tastes, so they aren’t as similar to each other. For example, Alice and Eve may both like horror movies, but disagree on comedies.</li>
</ul>
<p>But there are various reasons we might want to use a user-based approach <em>in addition to</em> an item-based approach (say, a user hasn’t rated many items yet, but we can find similar users based on other types of data, such as browsing history; or, we want to predict user u’s rating on item i, but user u hasn’t rated any items similar to i), so let’s see if we can get around these limitations.</p>
<p>To get around the first limitation, we can project users into a lower-dimensional space (say, by using a singular value decomposition), where we can use a space-partitioning data structure (e.g., a kd-tree) or a nearest-neighbor algorithm (e.g., locality sensitive hashing) to find neighboring users.</p>
<p>To get around the second limitation – that a user u may be predictive of user v for some items, but less so for others – we incorporate item-item similarity into our weighting method. That is, when using the user-neighborhood model to predict user u’s rating on item i, we give higher weight to items similar to i, by choosing the weights to minimize $\min_w \sum_{j \neq i} s_{ij} \left( r_{uj} - \sum_{v \in N(u, i)} w_{uv} r_{vj} \right)^2,$ where the $s_{ij}$ are item-item similarities.</p>
<h2>Appendix: Shrinkage</h2>
<p>Parameter shrinkage is used a couple times in the paper, so let’s explain what it means.</p>
<p>Suppose that we want to estimate the probability of a coin. If we flip it once and see heads, then the maximum-likelihood estimate of heads is 1. But (as is typical for maximum-likelihood estimates), this is severe overfitting, and what we should do instead is shrink this maximum-likelihood estimate to a prior estimate of the probability of heads, say 1/2. (Note that shrinkage doesn’t necessarily mean decreasing the number, just moving it towards a prior estimate).</p>
<p>How should we perform this shrinkage? If our maximum-likelihood estimate of our parameter $\theta$ is $x$ and our prior mean is $\mu$, a natural estimation of $\theta$ is to use a weighted mean $\alpha x + (1 - \alpha)\mu$, where $\alpha$ is some measure of the degree of belief in our maximum likelihood estimate.</p>
<p>This weighted average approach has several interpretations:</p>
<ul>
<li>We can also view it as a shrinkage of our maximum likelihood estimate to our prior mean: $\alpha x + (1 - \alpha)\mu = x + (1 - \alpha) (\mu - x)$</li>
<li>We can also view it as a Bayesian posterior: if we use a prior $\theta \sim N(\mu, \tau)$ (where $\tau$ is the precision of our Gaussian, not the variance) and a conditional distribution $x | \theta \sim N(\theta, \tau_x)$, then the posterior mean of $\theta$ is $\theta = \frac{\tau_x}{\tau_x + \tau}x + \frac{\tau}{\tau_x + \tau}\mu,$ which is equivalent to the form above.</li>
</ul>
</div>
Prime Numbers and the Riemann Zeta Function2011-03-14T04:15:00+01:002011-03-14T04:15:00+01:00Edwin Chentag:blog.echen.me,2011-03-14:/2011/03/14/prime-numbers-and-the-riemann-zeta-function/
<div class="entry-content"><p>Lots of people know that the <a href="http://en.wikipedia.org/wiki/Riemann_hypothesis">Riemann Hypothesis</a> has <em>something</em> to do with prime numbers, but most introductions fail to say what or why. I’ll try to give one angle of explanation.</p>
<h1>Layman’s Terms</h1>
<p>Suppose you have a bunch of friends, each with an instrument that plays at …</p></div>
<div class="entry-content"><p>Lots of people know that the <a href="http://en.wikipedia.org/wiki/Riemann_hypothesis">Riemann Hypothesis</a> has <em>something</em> to do with prime numbers, but most introductions fail to say what or why. I’ll try to give one angle of explanation.</p>
<h1>Layman’s Terms</h1>
<p>Suppose you have a bunch of friends, each with an instrument that plays at a frequency equal to the imaginary part of a zero of the Riemann zeta function. If the Riemann Hypothesis holds, you can create a song that sounds exactly at the prime-powered beats, by simply telling all your friends to play at the same volume.</p>
<h1>Mathematical Terms</h1>
<p>Let $ \pi(x) $ denote the number of primes less than or equal to x. Recall <a href="http://en.wikipedia.org/wiki/Prime_number_theorem#The_prime-counting_function_in_terms_of_the_logarithmic_integral">Gauss’s approximation</a>: $ \pi(x) \approx \int\_2\^x \frac{1}{\log t} \,dt $ (aka, the “probability that a number n is prime” is approximately $ \frac{1}{\log n} $).</p>
<p>Riemann improved on Gauss’s approximation by discovering an <em>exact</em> formula $ P(x) = A(x) - E(x) $ for counting the primes, where</p>
<ul>
<li>$ P(x) = \sum\_{p\^k < x} \frac{1}{k} $ performs a weighted count of the prime powers less than or equal to x. [Think of this as a generalization of the prime counting function.]</li>
<li>$ A(x) = \int\_0\^x \frac{1}{\log t} \,dt+ \int\_x\^{\infty} \frac{1}{t(t\^2 -1) \log t} \,dt $ $ - \log 2 $ is a kind of generalization of Gauss’s approximation.</li>
<li>$ E(x) = \sum\_{z : \zeta(z) = 0} \int\_0\^{x\^z} \frac{1}{\log t} \,dt $ is an error-correcting factor that depends on the zeroes of the Riemann zeta function.</li>
</ul>
<p>In other words, if we use a simple Gauss-like approximation to the distribution of the primes, the zeroes of the Riemann zeta function sweep up after our errors.</p>
<p>Let’s dig a little deeper. Instead of using Riemann’s formula, I’m going to use an equivalent version</p>
<p>$$ \psi(x) = (x + \sum\_{n = 1}\^{\infty} \frac{x\^{-2n}}{2n} - \log 2\pi) - \sum\_{z : \zeta(z) = 0} \frac{x\^z}{z} $$</p>
<p>where $ \psi(x) = \sum\_{p\^k \le x} \log p $. Envisioning this formula to be in the same $P(x) = A(x) - E(x)$ form as above, this time where</p>
<ul>
<li>$ P(x) = \psi(x) = \sum\_{p\^k \le x} \log p $ is another kind of count of the primes.</li>
<li>$ A(x) = x + \sum\_{n = 1}\^{\infty} \frac{x\^{-2n}}{2n} - \log 2\pi $ is another kind of approximation to $P(x)$.</li>
<li>$ E(x) = \sum\_{z : \zeta(z) = 0} \frac{x\^z}{z} $ is another error-correction factor that depends on the zeroes of the Riemann zeta function.</li>
</ul>
<p>we can again interpret it as an error-correcting formula for counting the primes.</p>
<p>Now since $ \psi(x) $ is a step function that jumps at the prime powers, its derivative $ \psi’(x) $ has spikes at the prime powers and is zero everywhere else. So consider</p>
<p>$$ \psi’(x) = 1 - \frac{1}{x(x\^2 - 1)} - \sum\_z x\^{z-1} $$</p>
<p>It’s well-known that the zeroes of the Riemann zeta function are symmetric about the real axis, so the (non-trivial) zeroes come in conjugate pairs $ z, \bar{z} $. But $ x\^{z-1} + x\^{\bar{z} - 1} $ is just a wave whose amplitude depends on the real part of z and whose frequency depends on the imaginary part (i.e., if $ z = a + bi $, then $ x\^{z-1} + x\^{\bar{z}-1} = 2x\^{a-1} cos (b \log x) $), which means $ \psi’(x) $ can be decomposed into a sum of zeta-zero waves. Note that because of the $2x\^{a-1}$ term in front, the amplitude of these waves depends only on the real part $a$ of the conjugate zeroes.</p>
<p>For example, here are plots of $ \psi’(x) $ using 10, 50, and 200 pairs of zeroes:</p>
<p><a href="http://dl.dropbox.com/u/10506/blog/riemann-hypothesis/10.png"><img src="http://dl.dropbox.com/u/10506/blog/riemann-hypothesis/10.png" alt="10 Pairs" /></a></p>
<p><a href="http://dl.dropbox.com/u/10506/blog/riemann-hypothesis/50.png"><img src="http://dl.dropbox.com/u/10506/blog/riemann-hypothesis/50.png" alt="50 Pairs" /></a></p>
<p><a href="http://dl.dropbox.com/u/10506/blog/riemann-hypothesis/200.png"><img src="http://dl.dropbox.com/u/10506/blog/riemann-hypothesis/200.png" alt="50 Pairs" /></a></p>
<p>So when the Riemann Hypothesis says that all the non-trivial zeroes have real part 1/2, it’s hypothesizing that the non-trivial zeta-zero waves have equal amplitude, i.e., they make equal contributions to counting the primes.</p>
<p><strong>In Fourier-poetic terms</strong>, when Flying Spaghetti Monster composed the music of the primes, he built the notes out of the zeroes of the Riemann zeta function. If the Riemann Hypothesis holds, he made all the non-trivial notes equally loud.</p>
</div>
Topological Combinatorics and the Evasiveness Conjecture2011-03-14T04:15:00+01:002011-03-14T04:15:00+01:00Edwin Chentag:blog.echen.me,2011-03-14:/2011/03/14/topological-combinatorics-and-the-evasiveness-conjecture/
<div class="entry-content"><p>The Kahn, Saks, and Sturtevant approach to the Evasiveness Conjecture (see the original paper <a href="http://www.springerlink.com/index/R521072311641L41.pdf">here</a>) is an epic application of pure mathematics to computer science. I’ll give an overview of the approach here, and probably try to add some more information on the problem in other posts.</p>
<p><strong>tl;dr …</strong></p></div>
<div class="entry-content"><p>The Kahn, Saks, and Sturtevant approach to the Evasiveness Conjecture (see the original paper <a href="http://www.springerlink.com/index/R521072311641L41.pdf">here</a>) is an epic application of pure mathematics to computer science. I’ll give an overview of the approach here, and probably try to add some more information on the problem in other posts.</p>
<p><strong>tl;dr</strong> The KSS approach provides an algebraic-topological attack to a combinatorial hypothesis, and reduces a graph complexity problem to a problem of contractibility and (not) finding fixed points.</p>
<p>First, the Evasiveness Conjecture states that any (non-trivial) monotone graph property is evasive. In other words, if you’re trying to figure out whether an undirected n-vertex graph satisfies a certain property (e.g., whether the graph contains a triangle or is connected), and this property is monotone (meaning that if you add more edges to the graph, then it still satisfies the property), then if all you’re allowed to do is ask questions of the form “Is edge (i, j) in the graph?”, then you need to query for every single edge before you can determine whether the graph satisfies the property or not. For example, if you want to figure out whether a graph G contains a clique of size 5, then you need to know whether each of the n(n-1)/2 possible edges is in the graph or not before you can answer for certain.</p>
<p>Next, given any monotone graph property on n-vertex graphs, we can associate it with a simplicial complex S (basically, an n-dimensional structure formed by gluing together a bunch of hypertriangles), by taking the complex to be the set of all n-vertex graphs that don’t satisfy the property.</p>
<p>Kahn, Saks, and Sturtevant then prove that if a monotone graph property is not evasive, then its associated simplicial complex is contractible, and thus (by the Lefschetz Fixed-Point theorem) any auto-simplicial map on the complex (a function from the complex to itself that preserves faces) has a fixed point.</p>
<p>Thus, we can prove that a monotone graph property is evasive by finding a simplicial map that has no fixed point (which we can do by showing that no orbit of the map is a face of the complex). This approach has been used to prove things like the evasiveness of graph properties when the number of vertices is prime or a prime power, and the evasiveness of all bipartite graph properties.</p>
</div>
Item-to-Item Collaborative Filtering with Amazon's Recommendation System2011-02-15T04:15:00+01:002011-02-15T04:15:00+01:00Edwin Chentag:blog.echen.me,2011-02-15:/2011/02/15/item-to-item-collaborative-filtering-with-amazons-recommendation-system/
<div class="entry-content"><h1>Introduction</h1>
<p>In making its product recommendations, Amazon makes heavy use of an item-to-item collaborative filtering approach. This essentially means that for each item X, Amazon builds a neighborhood of related items S(X); whenever you buy/look at an item, Amazon then recommends you items from that item’s neighborhood …</p></div>
<div class="entry-content"><h1>Introduction</h1>
<p>In making its product recommendations, Amazon makes heavy use of an item-to-item collaborative filtering approach. This essentially means that for each item X, Amazon builds a neighborhood of related items S(X); whenever you buy/look at an item, Amazon then recommends you items from that item’s neighborhood. That’s why when you sign in to Amazon and look at the front page, your recommendations are mostly of the form “You viewed… Customers who viewed this also viewed…”.</p>
<h1>Other approaches.</h1>
<p>The item-to-item approach can be contrasted to:</p>
<ul>
<li><strong>A user-to-user collaborative filtering approach</strong>. This finds users similar to you (e.g., it could find users who bought a lot of items in common with you), and suggest items that they’ve bought but you haven’t.</li>
<li><strong>A global, latent factorization approach</strong>. Rather than looking at individual items in isolation (in the item-to-item approach, if you and I both buy a book X, Amazon will make essentially the same recommendations based on X, regardless of what we’ve bought in the past), a global approach would look at all the items you’ve bought, and try to detect properties that characterize what you like. For example, if you buy a lot of science fiction books and also a lot of romance books, a global-approach algorithm might try to recommend you books with both science fiction and romance elements.</li>
</ul>
<h1>Pros/cons of the item-to-item approach:</h1>
<ul>
<li><strong>Pros over the user-to-user approach</strong>: Amazon (and most applications) has many more users than items, so it’s computationally simpler to find similar items than it is to find similar users. Finding similar users is also a difficult algorithmic task, since individual users often have a very wide range of tastes, but individual items usually belong to relatively few genres.</li>
<li><strong>Pros over the factorization approach</strong>: Simpler to implement. Faster to update recommendations: as soon as you buy a new book, Amazon can make a new recommendation in the item-to-item approach, whereas a factorization approach would have to wait until the factorization has been recomputed. The item-to-item approach can also be more easily leveraged in several areas, not only in the recommendations made to you, but also in the “similar items/other customers also bought” section when you look at a particular item.</li>
<li><strong>Cons of the item-to-item approach</strong>: You don’t get very much diversity or surprise in item-to-item recommendations, so recommendations tend to be kind of “obvious” and boring.</li>
</ul>
<h1>How to find similar items</h1>
<p>Since the item-to-item approach makes crucial use of similar items, here’s a high-level view of how to do it. First, associate each item with the set of users who have bought/looked at it. The similarity between any two items could then be a normalized measure of the number of users they have in common (i.e., the Jaccard index) or the cosine distance between the two items (imagine each item as a vector, with a 1 in the ith element if user i has bought it, and 0 otherwise).</p>
</div>