Kieran Healy

Updates to the Social Science Starter Kit

The Emacs Social Science Starter Kit is a drop-in collection of packages and settings for Emacs 24 aimed at people like me: that is, people doing social science data analysis and writing, using some combination of tools like R, git, LaTeX, Pandoc, perhaps some other programming languages (e.g., Python, or Perl), and plain-text formats like Markdown, and Org-Mode. More information on the kit is available here. Some of its highlights are listed here. It was originally written to accompany a more general article on Choosing Your Workflow Applications. The SSSK is available on github.

I’ve made some changes and additions to the kit recently, initially prompted by the release of Version 8 of Org-Mode. This was a major release that included a completely rewritten exporter. It is incompatible with the master branch of the Starter Kit. However, because Org-Mode 8 is not yet in the stable release of Emacs 24, I have left the SSSK’s master branch as-is, and created a new orgv8 branch. This is where new things are being added, and I am no longer adding features or making fixes to the master branch. When Org-Mode 8 gets into Emacs, the orgv8 branch will become the master branch. New things on the orgv8 branch include:

  1. The newest versions of ESS, Magit, and other packages previously in the kit.

  2. Cleaned up LaTeX exporting from Org-Mode. It now uses XeLaTeX only, and is much simpler than before. As with the earlier setup, you will need my collection of LaTeX Templates and Styles for it to work properly.

  3. Added Smartparens, Polymode, Powerline, support for previewing in Marked, Visual-Regexp, and better support for Zenburn and Anti-Zenburn themes.

  4. Python support is better now.

  5. Various clean-up and compatibility updates to accompany the new additions and updates, especially having to do with the many recent improvements to ESS. Autocomplete mode and ESS now work much better together.

Looking at plain text more generally, I’ve been trying to take advantage of Knitr as much as possible, as it seems to me to be the future of literate programming in R, particularly in conjunction with Markdown. Support for .Rmd files in ESS is not quite there yet, but Polymode provides a workable solution right now. The main benefits of working with Knitr and .Rmd is that it is very easy to get to both a good HTML file (i.e., a .md file that is easily previewable in Marked) and a good PDF file (via LaTeX). Some inevitable limitations remain, e.g. with the lack of cross-reference and certain limits to citations. But being able to combine R and Markdown is a clear benefit, given the prevalence of the .md format and the ease of export to other formats from there (via Pandoc). I should probably write a more thorough post about this at some point. In the meantime, enjoy the Kit.

Google Glass and the Need for XU Design

I was reminded this morning of an old Dotcom Era commercial from IBM. With some helpful prompting on Twitter, I eventually tracked it down. As you can see—pixelated video notwithstanding—IBM had some of the main concepts of Google Glass covered back in 2000, notably the clear presentation of the wearer as a jerk.

One of the standard jobs in software development these days is UX Design. User Experience covers “any aspect of a user’s experience with a given system … addressing all aspects of a product or service as perceived by users.” Products like Google Glass make it clear that we should formalize the development process further to include what we can call “Experience of User”, or XU Design. The XU Designer’s job will be to assess and tweak how third parties experience the users of your product or service. Is the XU experience intrusive? Is it annoying? Do our product’s XU Metrics all point in the direction of “Christ, what an asshole?” As the XU specialty develops we can trace its history back to phenomena like people loudly using cellphones in public, or people talking to you while wearing headphones, and the various ways norms and tolerances developed for these practices, or failed to develop. Right now, though, it looks like Google Glass is shaping up to be the leading XU Design disaster of our time.

Now if you’ll excuse me, I need to trademark the term XU Design and start a consulting company.

Sociology Rankings and the Fabio Effect

When I posted the Sociology Department Rankings for 2013 I joked that Indiana made it to the Top 10 “due solely to Fabio mobilizing a team of role-playing enthusiasts to relentlessly vote in the survey. (This is speculation on my part.)” Well, some further work with the dataset on the bus this morning suggests that the Fabio Effect is something to be reckoned with after all.

The dataset we collected has—as best we can tell—635 respondents. More precisely it has 635 unique anonymized IP addresses, so probably slightly fewer actual people, if we assume some people voted at work, then maybe again via their phone or from home. Our 635 respondents made 46,317 pairwise comparisons of departments. Now, in any reputational survey of this sort there is a temptation to enhance the score of one’s own institution, perhaps directly by voting for them whenever you can (if you are allowed) or more indirectly by voting down potential peers whenever you can. For this reason some reputational surveys (like the Philosophical Gourmet Report) prohibit respondents from voting for their employer or Ph.D-granting school. The All our Ideas framework has no such safeguards, but it does have a natural buffer when the number of paired comparisons is large. One has the opportunity to vote for one’s own department, but the number of possible pairs is large enough that it’s quite hard to influence the outcome.

It’s not impossible, however. The distribution of votes across our 635 respondents has a very long tail. While 75 percent of respondents registered just over 70 votes before finding something better to do, and 95 percent of respondents were done after about 250 votes, a brave few carried on for much longer.

"Voter histogram."

As you can see, a small number of respondents cast more than 500 votes, and a few lonely souls cast more than a thousand. I found myself wondering whether these few extreme cases materially affected the final rank order. And in the course of answering that question I found what might fairly be described as somewhat suspicious voting patterns in several—but notably not all—of our Supervoters. In particular, the respondent with the very largest number of votes (1425 in total) had two favorite departments. He or she voted on them multiple times in separate contests—more than thirty times apiece—and both departments won every time. (By chance, this voter was never presented with the two in a head-to-head contest.)

Now, it’s possible for this to happen quite straightforwardly: the departments that emerge at the top of the overall ranking are by definition the ones that win all or nearly all of their head-to-head contests. And given the range of disagreement about which departments should win, as reflected in the error bars around the point estimates, there are quite a few such departments. However, in the case of our Supervoter, the favored departments were somewhat further down in the final rankings, making their 100 percent winning streak seem a little odd, particularly given that no other voters shared their view in several high-disparity cases.

To preserve the confidentiality of the voting process—and bearing in mind that I do not have any identifying information whatsoever about the Supervoter in question—I will refer to the improbably favored departments by the pseudonyms “Hoosier University at Flowerington” and “The Fighting University of Our Lady”, and name the associated Supervoter phenomenon “The Fabio Effect”. Two other Supervoters also displayed somewhat suspicious voting patterns, showing a uniform but perhaps difficult-to-justify preference for Cornelius College and Twin-Citiversity, respectively, over all-comers.

The original rankings still stand, all the same, if only because the original error-ranges easily cover the shuffling around that happens once you re-run the model with the offending Supervoters removed. But I thought it would be worth showing what the rank-order looks like when the Fabio Effect is accounted for.

"Fabio-Adjusted Rankings."

Sociology Department Rankings for 2013

Last week we launched the OrgTheory/AAI 2013 Sociology Department Ranking Survey, taking advantage of Matt Salganik’s excellent All Our Ideas service to generate sociology rankings based on respondents making multiple pairwise comparisons between department. That is, questions of the form “In your judgment, which of the following is the better Sociology department?” followed by a choice between two departments. Amongst other advantages, this method tends to get you a lot of data quickly. People find it easier to make a pairwise choice between two alternatives than to assign a rating score or produce a complete ranking amongst many alternatives. They also get addicted to the process and keep making choices. In our survey, over 600 respondents made just over 46,000 pairwise comparisons. In the original version of this post I used the Session IDs supplied in the data, forgetting that the data file also provides non-identifying (hashed) IP addresses. I re-ran the analysis using voter-aggregated rather than session-aggregated data, so now there is no double-counting. The results are a little cleaner. Although the All Our Ideas site gives you the results itself, I was interested in getting some other information out of the data, particularly confidence intervals for departments. Here is a figure showing the rankings for the Top 50 departments, based on ability scores derived from a direct-comparison Bradley-Terry model.

"Top 50."

The model doesn’t take account of any rater effects, but given the general state of the U.S. News ranking methodology I am not really bothered. As you can see, the gradation looks pretty smooth. The first real “hinge” in the rankings (in the sense of a pretty clean separation between a department and the one above it) comes between Toronto and Emory. You could make a case, if you squint a bit, that UT Austin and Duke are at a similar hinge-point with respect to the departments ranked above and below them. Indiana’s high ranking is due solely to Fabio mobilizing a team of role-playing enthusiasts to relentlessly vote in the survey. (This is speculation on my part.)

You can do other things with this data, too. Here are the results of a cluster analysis of the votes, which brings out some interesting similarities.

"Cluster analysis."

Finally, Baptiste Coulmont carried out his own pairwise-comparison survey of French sociology departments—at least until he was overwhelmed by the pressure exerted by people who thought the very idea of such a ranking was morally offensive—and presented a nice analysis of it on his blog. Inspired by that, here’s are the results of a principal components analysis of the voting patterns.

"Factor analysis."

The x-axis is more or less the overall ranking itself. The y-axis is harder to interpret. The colors in the plot show how long people typically to vote for that department in comparisons, with Slow and Fast categories representing roughly the bottom and top quintles of the distribution of times. High-ranking but “Slow” departments are interesting here, as it suggests they might be difficult to place with respect to their peers.