TechnoCalifornia

Recsys 2011 - Notes and Pointers

2011-11-02T21:29:00.000-07:00

I found Recsys this year of very high quality in general. There were many good papers and presentations. The Industry track was also very high-quality, with very interesting talks from companies such as Twitter, Facebook, or eBay. Jon Sanders and I also gave two presentations explaining how recommendations have evolved since the Netflix Prize (more on this soon).

Here are my rough notes with pointers to some papers I considered especially interesting. I have grouped them in 5 categories that I think summarize the main topics in the conference: (1) Transparency and explanations, (2) Implicit Feedback, (3) Context, (4) Metrics and evaluation, and (5) Others. Note that the selection is completely biased towards my personal interests.

(1) TRANSPARENCY & EXPLANATIONS. One of the recurring themes was the fact that user trust and perceived quality of the recommendations was very much influence not by accuracy alone, but by how transparent the system was, and the amount of "explanations" that were added.

Daniel Tunkelang(LinkedIn) did a very interesting tutorial on "Recommendations as a Conversation with the User", where he focused on these kinds of issues. See his slides in his blog.
Neel Sundaresan (eBay) also stressed in his keynote that adding explanations can sometimes be more important than getting the recommendation right.
In the paper "Each to His Own: How Different Users Call for Different Interaction Methods in Recommender Systems", the authors found that depending on how experts are users in the domain, they prefer different kind of recommendations and interaction models. For example, in one of the extremes, novices, prefer top-10 non-personalized to their personalized recommendations. In general a hybrid model of interaction is better than either implicit or explicit-only.

(2) IMPLICIT FEEDBACK. A lot of papers this years on using implicit consumption data instead of (or in combination with) ratings.

The best paper, by Yehuda Koren and Joe Sill, addressed the issue of non-linearity in ratings. "OrdRec: An Ordinal Model for Predicting Personalized Item Rating Distributions" modifies the standard Matrix Factorization approach to adapt to the fact that user ratings are ordinal, but not numerical. The way they model ratings, with a set of thresholds, can be used in combination with any model, not only SVD-like approaches. This paper effectively addresses most of the issues I raised in my previous post "We are doing everything wrong..."
In "Modeling Item Selection and Relevance for Accurate Recommendations: A Bayesian Approach" they define the concept of a "Free probabilstic model" where they try to predict independently the probabilty of play and rating.
In "Multi-Value Probabilistic Matrix Factorization for IP-TV Recommendations", the authors present a Matrix Factorization model that allows for multiple observations of the same item. In particular, it is applied for IPTV recommendations where the fact that the user watched part of an episode is interpreted as negative feedback.
"Matrix Co-factorization for Recommendation with Rich Side Information and Implicit Feedback" presents a combined Matrix Factorization model that includes ratings, content features, and implicit feedback. They use cosine item similarity for weighing negative examples.
In "Personalizing Tags: A Folksonomy-like Approach for Recommending Movies", they use tags (or categories) as a very simple method of recommending movies: for each user compute average rating given to movies with a certain tag.

(3) CONTEXT. There were 2 workshops (CARS and CAMRA), and several papers in the main conference, talking about how to add contextual information for the recommendations:

"The Effect of Context-Aware Recommendations on Customer Purchasing Behavior and Trust" is an interesting paper, focusing on the evaluation side. They include an A/B test for measuring the effect of context-aware recommendations. Using context increased overall sales in $ but not in number. Therefore, users tend to spend more $ per item.
In the CAMRA workshop, many papers (such as "Temporal Rating Habits: A Valuable Tool for Rater Differentiation" or "Identifying Users From Their Rating Patterns") were related to how to identify who the author of a rating in a household was, since this was one of the tasks for the contest.
Also related to group recommendations, "Group Recommendation using Feature Space Representing Behavioral Tendency and Power Balance among Members", tries to model what is a good recommendation for a group where each of the individuals does not have the same influence.

(4) METRICS and EVALUATIONS: There were several papers that offered different ways to measure accuracy for top-N ranked recommendations.

"Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems" presents an interesting framework that includes metric for measuring not only accuracy, but also novelty, diversity....
"Item Popularity and Recommendation Accuracy" is an interesting work on how to remove popularity bias from accuracy metrics. A user study validates the fact that recall measure is correlated with user perceived quality of recommendation. Besides proposing a recall metric that removes popularity bias, he also proposes a popularity stratified training method that weights negative examples according to how popular they are.
"Evaluating Rank Accuracy based on Incomplete Pairwise Preferences" proposes a measure called expected discounted rank correlation for the specific case of implicit feedback.

(5) OTHERS

eBay and UCSC presented "Utilizing Related Products for Post-Purchase Recommendation in E-commerce". The paper won the best poster award
There were many papers on Social Recommendations. Just to name one, in "Power to the People: Exploring Neighbourhood Formations in Social Recommender Systems", they did a user study to figure out how much users would like and trust recommendations coming from different user groups (those they decided, friends, everyone...). Interestingly, the method of choice did not make much difference... until you told the users what it was.
In "Wisdom of the Better Few: Cold Start Recommendation via Representative based Rating Elicitation" they discussed how to select most imformative users and items for cold start. I was surprised to see that our "Wisdom of the Few" approach got paraphrased in a paper title.
There were a couple of very interesting workshops on Music Recommendations and Mobile Recommendations that I had to miss since I was attending others. But, they are definitely worth looking into if you are into music or mobile.

The Recommender Problem & the Presentation Context

2011-09-25T23:41:00.001-07:00

In the traditional formulation of the "Recommender Problem", we have pairs of items and users and user feedback values for very few of those dyads. The problem is formulated as the finding of a utility function or model to estimate the missing values.

In many real-world situations, feedback will be implicit** and binary in nature. For instance, in a web page you will have users visiting a url, or clicking on an add as a positive feedback. In a music service, a user will decide to listen to a song. Or in a movie service, like Netflix, you will have users deciding to watch a title as an indication that the user liked the movie. In these cases, the recommendation problem becomes the prediction of the probability a user will interact with a given item. There is a big shortcoming in using the standard recommendation formulation in such a setting: we don't have negative feedback. All the data we have is either positive or missing. And the missing data includes both items that the user explicitly chose to ignore because they were not appealing and items that would have been perfect recommendations but were never presented to the user.

A similar issue has been dealt with in traditional data mining research, where classifiers need to be trained only using positive examples. In the "Learning Classifiers from Only Positive and Unlabeled Examples" SIGKDD 08 paper, the authors present a method to convert unlabeled examples into both a positive and a negative example, each with a different weight related to the probability that a random exemplar is positive or negative. Another solutions to this issue is presented in the "Collaborative Filtering for Implicit Feedback Datasets" paper by Hu, Koren and Volinsky. In this work, the authors binarize the implicit feedback values: any feedback value greater than zero means positive preference, while any value equal to zero is converted to no preference. A greater value in the implicit feedback value is used to measure the "confidence" in the fact the user liked the item, but not in measuring "how much" the user liked it. Yet another approach to inferring positive and negative feedback from implicit data is presented in the paper I co-authored with Dennis Parra, and I presented in a previous post. There, we argue that implicit data can be transformed to positive and negative feedback if aggregated at the right level. For example, the fact that somebody listened only once to a single track in an album can be interpreted as the user not liking that album.

In many practical situations, though, we have more information than the simple binary implicit feedback from the user. For unlabeled examples that the user did not directly interact with, we can expect to have other information. In particular, we might be able to know whether they were shown to the user or not. This adds very valuable information, but slightly complicates the formulation of our recommendation problem. We now have three different kinds of values for items: positive, presented but not chosen, and not presented. And this is only if we simplify the model. In reality, information related to the presentation can be much richer than this and we might be able to derive data like the probability the user actually saw the item or weigh in different interaction events such as mouse overs, scrolls...

In Netflix, we are working on different ways to add this rich information related to presentations and user interaction to the recommender problem. That is why I was especially interested in finding out that this year's SIGIR best student paper award has been awarded to a paper that addresses this issue. In the paper "Collaborative Competitive Filtering: Learning
Recommender Using Context of User Choice", the authors present an extension to traditional Collaborative Filtering by encoding into the model not only the collaboration between similar users and items, but also the competition of items for user attention. They derive the model as an extension to standard latent factor models by taking into account the context in which the user makes the decision. That is, the probability I decide to select a given item depends on which are the other items I have as an alternative. Results are preliminary but promising. And, this work is definitely an interesting and appealing starting point for an area with many practical applications.

However, there are many possible improvements to the model. One of them, mentioned by the authors, is the need to take into account the so-called position bias. An item that is presented in the first position of a list has many more possibilities to be chosen than one that is farther down. This effect is well-known in the search community and has been studied from several angles. I would recommend, for instance to read some of the very interesting papers on this topic by Thorsten Joachims and his students. In the paper “How Does Clickthrough Data Reflect Retrieval Quality?”, for instance, they show how arbitrarily swapping items in a search result list has almost no effect. This proves that the positioning of the element can be a most important factor than how relevant the item is.

I would love to hear of other ideas or approaches to deal with this new version of the recommender problem that includes, and would encourage researchers in the area to address an issue of huge potential impact.

**Note: I am using the word implicit here in the traditional sense in the recommendation literature. The truth is that a user selecting an item is in fact explicit information. However, it can be considered implicit in that the user is informing about the preferences indirectly by comparing the item to others in a context.

Joining Netflix

2011-07-28T21:49:00.000-07:00

Three weeks ago, I started to work for Netflix. Everything has moved so fast with so many things to do and learn that it seems like I have already been here for a much longer time!

I am now working as the manager of a small team working on recommendations & personalization in the company that promoted recommender systems research to major headlines thanks to the Netflix Prize. It also feels great to come to the company in an exciting time when it has just reached its 25th million customer and is starting its international expansion to Latin America.

All the fuzz created around the Netflix Prize might lead some to believe that rating prediction is all there is to Netflix suggesting a given movie. However, I was happy to find out that rating prediction is only one of the many signals that my team uses in creating the final suggestions.

Awesome place, awesome people, and awesome time to be around. And, btw, we are hiring, so let me know if you are interested in joining. (Update: it seems that the jobs link is currently not active outside US/CA... I'm working on getting this fixed)

Walk the Talk: On the Combination of Implicit and Explicit Feedback

2011-07-18T22:32:00.000-07:00

Last week, Denis Parra presented our paper entitled "Walk the Talk: Analyzing the Relation between Implicit and Explicit Feedback for Preference Elicitation" at the UMAP conference. The paper won Denis the best-student paper award (Congratulations!).

The paper presents our initial work in analyzing the relation between implicit and explicit feedback. In short, the main question we wanted to answer is how does the self-reported preferences users give in a typical 5-star interface relate to what they actually do when looking at their consumption patterns. Our hypothesis was that there should exist simple models that relate both kinds of feedback. Finding a way to robustly convert implicit feedback into explicit ratings would open up the door to applying well-known methods with implicit feedback. But, much more importantly, we could then combine both kinds of input in a single model.

In order to test our hypothesis, we prepared an experiment in the music domain. We asked last.fm users to take a survey in which we queried them about how much they liked albums that were already in their listening history. With this data in hand, we could analyze the relation between implicit and explicit feedback and try to fit a simple model.

I recommend you read the full paper if you want to get the longer story of our findings, but here is a brief summary:

There is a strong correlation between implicit feedback and self-reported preference (see figure below)
Variables such as recentness of interaction or overall popularity do not have significant effect. Note that in a previous study by Salganik & Duncan Watts, global popularity was found to affect users perceived quality. However, in that case and as opposed to ours, users were made aware of the popularity.
Interaction effect: When listening to music, some people prefer to listen to isolated songs or albums. The way they interact with music, affects the way they report their taste.

After our analysis, we then construct a linear model that takes into account these variables by performing a linear regression. Once we have built these models, we can evaluate their performance in a regular recommendation scenario by measuring the error in predicting ratings in a hold-out dataset.

This paper represents an initial but very promising line of work that we have already improved in several ways such as the use of logistic instead of linear regression to account for the non-linearity of the rating scale or the use of the regression model as a way to combine both implicit and explicit feedback. But I will leave those findings for a future post.

Recommender Systems: We're doing it (all) wrong

2011-04-07T13:59:00.000-07:00

A few days back, there was an interesting post by Judy Robertson in the Communications of the ACM blog. The post, entitled "Stats: We're doing it wrong", builds upon a paper from last year's CHI conference in which they report that more than 90% of the HCI researchers used the wrong statistical tools when analyzing and reporting on likert scale type of data. A Likert scale is a unidimensional scale on which the respondent expresses the level of agreement to a statement - typically in a 1 to 5 scale in which 1 is strongly disagree and 5 is strongly agree.

Here is an excerpt from the post that I think is worth highlighting:

Likert scales give ordinal data. That it (sic), the data is ranked "strongly agree" is usually better than "agree." However, it's not interval data. You can't say the distances between "strongly agree" and "agree" would be the same as "neutral" and "disagree," for example. People tend to think there is a bigger difference between items at the extremes of the scale than in the middle (there is some evidence cited in Kaptein's paper that this is the case). For ordinal data, one should use non-parametric statistical tests which do not assume a normal distribution of the data. Furthermore, because of this it makes no sense to report means of likert scale data--you should report the mode.

As Judy, I have to admit that I am not a stats expert myself either. But in the general case I would agree with the previous: likert scale data is ordinal and cannot be treated as interval. However, whether treating it as interval is always a mistake or can be accepted under some circumstances is something that I am not sure and relates to the rest of this post.

So for instance, it is not uncommon to find references where they clearly state that likert data can be treated as interval. For example, look at what they say in this handbook edited by the FAO.

Likert scales are treated as yielding Interval data by the majority of marketing researchers.

Or look at the answer to the question of whether likert data can be treated as interval in stackexchange.

So there might be some circumstances in which, depending on the analysis, likert could be treated as interval... I guess. But not in the general case.

Implications for Recommender Systems

Now onto the big question: What does this have to do with Recommender Systems and how does it affect? To start with, let me ask you the question: Does the likert (1 to 5) scale relate to anything we use in recommender systems? You got it: ratings !

So our worry goes now to understanding whether ratings can be treated as interval or they should instead be treated as ordinal data, just as they are in the general case of the likert scale. In order to defend that ratings can be treated as interval, we should have some validation that the distance between different ratings is approximately equal. However, just as in the case of likert scales, we know this is not the case.

Look at this figure from our previous work on measuring noise in ratings.
Here we are plotting the probability of finding different kinds of inconsistencies between pairs of ratings. The probability that a user changes her rating between 2 and 3 is almost 0.35 while the probability she changes between 4 and 5 goes down to almost 0.1. This is a clear indication that users perceive that the distance between a 2 and a 3 is much lower than between a 4 and a 5.

Consequences

At this point, we can safely say that ratings are ordinal but not interval data. However, they are treated as a continuous interval scale in most of the recommender systems research! Let us stop to think a few of the consequences of ratings not being interval data.

Distance Measures: All the neighbor based methods in collaborative filtering are based on the use of some sort of distance measure. The most commonly used are Cosine distance and Pearson Correlation. However, both these distances assume a linear interval scale in their computations! We should conclude that using these distance measures with rating data is wrong. Other measures such as Spearman's rank correlation, do not assume this. But to be honest, I don't remember having read many papers using Spearman.

Error Measures: This is my favorite one... The most commonly accepted measure of success for recommender systems is the Root Mean Squared Error (RMSE). But wait, this measure is explicitly assuming that ratings are also interval data! Similar error measures such as MAE also fall in the same trap... banned! So what could we use? Standard Information Retrieval measures such as Precision and Recall do not necessarily assume interval scale on the ratings, although their mapping to recommendation efficiency may also be questioned. Rank-based measures such as Discounted Cumulative Gain (nDCG) seem like our best bet for now.

Matrix factorization: Most MF techniques in Recommender Systems are in fact optimizing for RMSE. Therefore, we should discard them as statistically incorrect for the same reasons stated above. There are interesting alternatives to this though, like the PureSVD method presented in Recsys last year, that do not optimize for RMSE but rather for ranking.

Conclusion

It is clear that explicit ratings, just like likert scale data, have to be treated like ordinal (and not interval data). However, most of the methods and measures currently in use in recommender systems assume in some sense that there is a continuous linear scale in the ratings. Of course I am not advocating for throwing all of this research to the trash (among other things, it would include much of mine), but I would advice for a drastic change in the way we approach these issues.

I am writing this post, especially in the hope to get feedback and reactions from you. So I am looking forward to the comments.

Update: This post was featured in Ycombinator Hacker News. So far it has received over 6K views and there is a somewhat interesting comment thread in Ycombinator.

(I'd like to thank and acknowledge the contribution of Denis Parra, Alexandros Karatzoglou, and Rodrigo Oliveira to this post through previous very fruitful discussions)

The Science and the Magic of User Feedback

2011-03-18T08:25:00.000-07:00

That was the main title of a series of talks I gave in different labs and companies during my recent California tour. In this presentation, I talked about many of our recent projects related to how to interpret user feedback, in general, and in the particular case of recommender systems. I talked about our work on measuring user rating noise, our follow-up in devising algorithms to reduce this natural noise, and on how you can use experts instead of crowds to not only minimize this noise but address other issues in collaborative filtering.

I also gave a sneak preview of our results to the music survey I announced some time ago. Denis Parra and I have submitted this work recently and are hoping to get it accepted to tell you a bit more about how to map implicit to explicit feedback.

The Science and the Magic of User Feedback for Recommender Systems

Update: Thanks to the guys at LinkedIn's SNA group, I have now added below the video of my presentation at LinkedIn... enjoy!

Tech Talk: Xavier Amatriain (Telefonica) -- "The Science and Magic of User and Expert Feedback for Improving Recommendations" from Talks at LinkedIn on Vimeo.

Managing Research the Agile Way

2011-03-15T03:56:00.000-07:00

I have discussed previously on this blog about how well the Scientific Method adapts to Agile approaches. These ideas also took me to an unfinished effort to draft an Agile Research Manifesto. However, by talking to several people with similar ideas, I realized that these attempts were largely interpreted as an intellectual exercise with little practical application. It is clearly my fault for not having explained that all of this in reality comes from many practical experiences. Some of these experiences go back to my PhD years when managing the CLAM framework, as well as many undergrad student projects. As a matter of fact, during those days I published a practical guide for students on how to do their final project the "agile way" (I still keep the webpage, in catalan, for historical reasons).

In any case, in this post I wanted to address the practical side of agile research management by giving you a flavor of how I try to manage projects.

(Picture by ~Tone)

The anatomy of a research project

What am I talking about when I say a "research project"? Although they might be completely different in theme and even scope, all of the projects that I have in mind when explaining the agile management approach should share at least some of the following properties:

Small-sized team: It is very likely that we are dealing with a one or two researchers team. A 3-4 people research team can already be considered large in my experience.
Very open and imprecise requirements: Especially at the beginning, we might have a coarse idea or hypothesis to validate. However, the approach, method, and scope, are likely to be undecided until very late in the game.
High risk: By definition, a research project has to be highly innovative and therefore... risky. Our goal is to minimize the cost of a failure and realize early on but not to remove failure since this is an intrinsic feature of risk.
Imprecise resources: The fact that requirements are not clear and risk is high is usually accompanied by the fact that resources that can be allocated to the project are usually imprecise. If the project is highly successful and proves its interest in the first iterations, it can grow into something larger with more resources added to it. On the other hand, it is also very likely to be killed quickly if it does not yield promising initial results.

The planning game

I will usually start-off by devoting a couple of weeks to a Sprint 0 during which the main tasks will be:

Understand what has been done before: Obviously, this requires lots of reading. However, it is good practice to also start writing at this same time, maybe in an informal wiki or the like.

Define the tools: Unless you are in a very specific environment, tools are likely to change for every project. Sometimes it is not only about what is the best tool, but also about what the team is most familiar with. This is usually an important thing in most projects, but it is more so in a project that it is high risk in nature and should avoid spending lots of time/resources in adapting to new tools.

Define the initial scope: There is no way you can have a complete picture of what is going to be the output of the project by this time. However, you should be able to list what you think will be the main steps and even some findings you anticipate. This list should be written like an ever changing Product Backlog (prioritized list of high-level features).

(Picture by Jenna Carver)

Prioritizing

One of the most important activities that you end up doing when planning any project, be it at the initial phase or at any of its iterations, is prioritizing the different requirements, stories... Doing this in a group meeting is a great way to gain insights on the project and to be strategic. Prioritizing tasks is not much different from any cost/benefit analysis: you measure cost, you measure benefit, and then sort items according to benefit/cost ratio.

In the case of project planning, I usually like to assign cost to "complexity", and benefit to "interest". In other words, the cost of a feature or story will be how difficult or complex we anticipate it is to implement it. And the benefit is how interesting or important it is for our final goal. Once you sort items using the interest/complexity ratio, you will find that easy-to-do yet interesting features float to the top, while complex and not so important sink down to the bottom.

Of course, the interesting discussions happen right in the middle. And especially when we have something that seems to be very important, but also very complex to achieve. In these cases, we feel tempted to jump right away at the problem and devote 100% of our energy to it. However, one of the agile principles is that things seem more complex when you don't have enough understanding. If you put them off to later iterations, they will eventually become clearer and clearer and end up surfacing to the first positions on your priority list. I have found this sort of smart procrastination to be extremely useful for agile research management.

Iterate, Iterate, Iterate

Once you have come up with your initial product backlog, it is all a matter of breaking the process down in short iterations - I usually plan for one-week. At the beginning of each iteration (or Sprint), you look at your product backlog, pick some of the top stories and break them down into finer grain tasks. You do the prioritization game on this new list and come up with your next week's scrum/iteration backlog.

When doing this finer-grain prioritization, I have found it very useful to use the estimated number of hours as the measure of "complexity". Therefore, when picking the top tasks of our list, we will also have an estimate of how feasible it is to have them during this iteration and how much will be the relative effort put into each of them. And, if any task is estimated to be more than a day long, do yourself a favor and break it into several tasks.

Also, it is important that, especially during the first iterations, you realize that the continuation of the project might be at stake at each iteration (or at least the current approach). Therefore, when measuring the "importance" of tasks to prioritize, ask yourself how relevant will that task be to convince you and others that you are onto something or you need to change routes.

Test-driven research

If you are familiar with agile methods, you will probably know how important testing is in an agile project. Tests not only guarantee the stability of the project but are actually a way to specify requirements in a more verifiable form. In a similar way, you can think of specifying many of your research hypothesis as a test. For example, you can turn your hypothesis that the effect of a given procedure on your data or population is significant by a verifiable assertion that for t-test(D_original, D_after_procedure)-> p is smaller 5%. There are many hypothesis and research tasks that can - and should - be written in this form before making it to your prioritized todo list. At least you should worry on how any of your results will be validated and how you can trust for them to be consistent and significant.

(Picture by Dunechaser)

Related approaches

If you are interested in this kind of approaches, I recommend you read the article on the SCORE method, which is somewhat related to many of the things I am mentioning here. Here you can read an interesting paper on doing test-driven research. Finally, I find the Pomodoro method a very interesting approach to individual time management. Since many research projects end up being quasi-individual, Pomodoro fits them pretty well.

On Trust Networks and Gamification. Or How Quora can overcome its Hype and embrace long-term Success

2011-01-31T13:26:00.000-08:00

If you are reading this blog I am pretty sure that you know quite a lot about Quora by now. If not, you should sign on and try it a bit before you continue reading the post.

I have to admit it, the first time I saw Quora I thought it looked like a watered-down version of stackoverflow , only with a much broader scope. The ability to follow was nice but... "big deal", I thought. However, I was missing the important point of the seamless integration between Quora and existing OSN, namely Twitter and Facebook. I always say that for an OSN to succeed it needs to ride on all the previous successful ones (including email if you allow me to stretch the definition of OSN that far), but I missed that part in quora until its hype began. Having quick connection to Twitter and Facebook, allowed Quora to overcome the always feared cold-start problem. You sign to Quora and in no time you are "connected" to all your "friends" and can start following their questions, their answers, votes... cool!

Well, so it seemed. But in no time, just as quick as people starting hyping about the service they were complaining about it and predicting its failure. This recent post at Techcrunch does a pretty good at summarizing and linking to the main Quora bitchmemes. Don't miss the original post by Vivek Wadhwa or some of the threads at Quora itself. You should also read this very illustrative piece on how Scobble went from love to hate in a matter of weeks.

To summarize, the two biggest complaints are the following: (1) Quora will inevitably be overtaken by spam and there will be no way to find good content anymore; and (2) producers of good content (answers) will become tired of the system and progressively leave making problem (1) even more inevitable.

While I do agree that these (and many other) issues are very important, I don't see them as inevitable and, in the following paragraphs, I would like to describe two ways to address them. But to start with, let me just state that believing that Quora can survive on being an inside-moderated network is not the answer. So what can be done?

Trust networks

In a trust network, nodes (users) have an associated trust value that is somehow used to decide how its contribution will be taken into account by the rest of users. For instance, in a recommender system, I can push content by neighboring nodes I trust while filtering out that coming from nodes with a lower trust value. In more sophisticated versions, trust is not a unique value but can be topic-specific. That is, my trust value can be very high for independent music but very low for classic literature. (If you are interested in the general topic of Trust and Social Networks you can read Golbeck's book or any of her many publications or presentations available online)

So let's go back to Quora now: why should they implement a trust network overlay? and, how could they implement a useful one? There are several reasons for why they should be doing so. But let us focus on the spam issue. You do not want for bad answers to get promoted by bad/evil users. The way around it is to not give these users the power to promote answers. And you can do this quite easily by assigning trust values to users. It would take 100 votes by "level 1" users to get an answer to the level of another one with just one vote by a "level 100" user. Of course, as I was mentioning before, this trust level could be topic-sensitive. Makes sense, doesn't it?

But, there a number of issues that are still unsolved on how to implement this trust network. The first one is who decides to promote and demote users? My answer is quite simple: users themselves. Whenever your answer gets voted up/down so would your trust level. And again, how much this level would go up/down would depend on the trust level of the voting user.

The only important remaining issue to such an approach is how to deal with the cold-start issue. But the answer to this would come from the integration to other OSN I was mentioning at the beginning. If I were implementing this kind of system, I would give users an initial trust level based on their TunkRank or their Klout Score.

Gamification

The other major issue that still needs to be tackled is how we guarantee that users do not become tired of the system and abandon it. I hope it is clear by now that the approach I described above would make things much more interesting for users interested in promoting their trust level. In fact, this is very close to what is known as gamification (see also game dynamics or game mechanics for very related concepts). Attach a badge to given levels of trust for some topics and you can start competing with Foursquare check-ins.

The use of badges, or game dynamics in general in Q&A sites is by no means new. Actually, stackoverflow, that I was referring to earlier in the post, delivers topical badges. And obtaining the first badge on a given topic can be an important accomplishment worth noting in your resume. But stackoverflow did not come up with this idea out of the blue: levels of expertise in forums have been used for a long time (see the Coffee Cups/Beans in Ubuntu forums, for instance).

I am not saying that implementing these two approaches would guarantee Quora's success. But not implementing them will probably guarantee the opposite. We have all seen potential in Quora. Apart from the quick integration with existing OSN and the ability to follow, there is the real-time component that brings it closer to a Q&A Twitter. If they don't fix these potential issues, somebody else will come up with an improved version that can very well be the "next big thing" that some social media gurus were seeing in Quora just some weeks ago.

Did you prepare your talk?

2010-11-16T16:12:00.000-08:00

I don't consider myself to be a great presenter. As a matter of fact, every time I finish a presentation, I find myself thinking about how many things I screwed up and could have done much better. However, whenever I attend a conference I face the cruel reality: my presentations are way better than most research presentations. If I am really not that good, it can only mean one thing: researchers generally suck at presenting their work. (This is in fact one of the reasons I am against organizing research conferences around oral presentations. But this is another discussion I will leave for another post).

So, if you have any doubts of whether you could be in that category of good researcher/poor presenter, you can do a quick test: Watch the video below. If you think your last presentation is well summarized in the video, you definitely fit into the group. Even if you don't, you might find some tips or advice of interest to you in the rest of this post.

OK, so what are the three basic rules to make a decent presentation? Easy: (1) Prepare yourself, (2) prepare yourself, and (3) prepare yourself.

At this point, you might already be tempted to stop reading because you disagree with what I am saying. I have found several reasons why people disagree with something as obvious as the fact that making a good presentation requires preparation, but I think all of them are summarized in the two following:

(a) I'm a natural: Maybe you are the kind of self-assured person that thinks that has great presentation skills and those shine best the more you improvise. I was pretty close to this myself some time ago. But if you fit into this category, there is a very easy test you can do: tape yourself on video on several presentations. If you still think you are great and need no preparation or further skills, congratulations! But chances are that then you realize how many things you have been doing wrong and how much you can improve. All great presenters I know stress the fact that preparation is key, period.

(b) I'm a researcher, not a TV Star: On the other extreme, you might be aware of your limitations but might think that this is not such a big deal. You are a researcher and live in the world of formulas, theories, or code. You could care less about what people get from your talks and you would be happy standing up and doing the chicken, chicken, chicken presentation. And this is not an exaggeration: I have seen junior researchers that are still editing slides a couple of hours before their scheduled presentation in a top conference. My take on this is the following: if you think presentation skills are not part of what is required in a researcher, you are wrong.

So by this point I will suppose that you are convinced of the importance of preparing research presentations. Ideally, you have also taped yourself and found that there are many things to improve. The question is what to do next. Obviously, I cannot pretend to summarize a presentation skills course in a post. There are thousands of resources out there in the form of books, videos, or similar that you will find without problem. But I do think that I can pinpoint a few issues that are important and tricks that might help.

First, I think it is important to separate two kinds of "preparation": (1) mid/long-term preparation aimed at improving your skills, and (2) short-term preparation for your next presentation.

Improving your skills

Again, you can find many books and resources on how to do this. But some of the things that you should at least consider:

(1) Tape yourself:

This will make you aware of the weak points and where you need to focus your efforts

(2) Enjoy the stage:

Some people have a really hard time every time they go onto stage, and this shows. There are many things you can do to learn techniques and improve on this that go from playing in a band to taking some acting and performance lessons (I did this and found it very useful and enjoyable)

(3) Read about it:

No need to become obsessed. But reading a couple of books or watching some videos giving you tips is not going to hurt. And remember, this is part of your expected skill set as a researcher. If you want a starting point, I can recommend you read a short 12 page essay on "How to give an academic talk v4.0" by Paul N. Edwards from U. Michigan.

(4) Rehearse the techniques:

It is very good if you have situations where you can rehearse what you learn from the previous. Actually, many of the techniques can be applied in "real" life (e.g. when talking to your boss). Others require of a more realistic setting. I have been lucky to use the courses at the university as a rehearse playground for improving my skills

(5) If you need help, look for it:

I have seen many cases of researchers with severe communication problems when presenting. Maybe I sound too harsh here, but I don't think this is acceptable. If you really want to be a researcher but don't think you can get to an acceptable level of presenting either (a) have some co-author present for you or (b) find some professional help. And this latter would be my preferred option. It is not so hard nowadays to find coaches or places that can help you out and if you agree this is an ability that you need in your job (and, again, you should agree), it is worth that you invest on it.

Preparing your next presentation

Regardless of whether you manage to improve your general presentation skills or not, you will have to face your next presentation sooner or later. When preparing the talk, you should focus on its two main components: the slides, and the talk itself.

The slides

Again, there are many resources out there on how to prepare your slides, the style, the design.. I was lucky to attend a course on Zen presentation style by the guys at Presentaciones Artesanas. Zen-style presentations are the kind of slides you will see at TED, for instance. I try to bear some of the techniques in mine but (a) I am not a professional presenter (that is, although presentations are important in a researcher's life, I have other things to do), and (b) sometimes, transmitting scientific rigor in a very graphical style is not easy (actually, according to Tufte, even Powerpoint should be banned from scientific publications). However, I do recommend to understand some of the design concepts behind the Zen style and maybe use some of them as a basis.

Once you have found your style, you will need to do the following tasks:

(1) Know your audience:

Before you start preparing the presentation, take some time to understand who you will be talking to. It's not the same to do a talk at a conference than pitch your work to business people, present to a prospective employer, or, like I did last week, try to convince high-schoolers of how cool Computer Science is.

Even if you are only focusing on research presentations at conferences, they are not all the same! Sometimes you will be giving a talk in a setting where everybody is an expert in what you are talking about, while in other occasions only a tiny fraction of the audience is working in your same field. In my case, I won't use the same kind of approach if I am presenting at a Recsys conference where everybody knows about Recommender Systems than at a generic one like WWW, where I can only assume that most of the audience does not know the topic in depth.

It is also important to look at the program schedule. The name of your session and the talk immediately before and after yours is going to give you more information about who might be sitting in. If you are presenting in a conference with multiple tracks, the talks scheduled at the same time as yours will give you some hint about who is *not* going to be attending yours.

(2) Find "the message"

Find a simple take-away message that you want to get through to your audience. In many cases it will be something along the lines of "look how important and interesting my research is, please go ahead and look more into it by reading the paper... and don't forget to cite it in your next publication". But in other to transmit that idea you need to make your point. Therefore, find the answers to: (a) what problem does your work solve, (b) what makes your work different from other solutions, and (c) why should anybody care about it. These three questions should help you find the message. Stick to one/two ideas and refer the audience to the paper for more details. Trying to squeeze in too many messages in too little time is a recipe for disaster.

Some researchers like to add another secondary message thread: (d) I am really smart and what I did is so complicated you might not even grasp it... I particularly dislike this kind of presentations and find them pretentious and boring (maybe because I am not so smart). But hey, I know some people have made quite a career of this so you should be aware.

(3) Prepare a script

Once you have identified the "main message," you are ready to prepare the script of the slides. I usually start off by having a bunch of empty slides with only the title on them. By having this, I can see if I might be going over time, need to sort things out differently... The script will depend on the kind of talk and time you have to speak. But in general, it will have a structure such as:

Introduce context and situation
Formulate problem and why it is important to solve
Main message (Solution to the problem, consequences, details on the solution...)
Summary on problem and solution
Future work and things to do

The script is important, but be ready and willing to change it. You are likely not to get it perfect from the start and as soon as you start adding more detail you will see a clearer picture. Don't make the "sticking to the plan" hit you back.

(4) Make the visuals

Maybe you think this is the least important part of your presentation. In my experience, I have come to value the visuals very much. Actually, most of the time I spend in preparing some presentations is looking for appropriate visuals that back up and re-enforce the "main message". The less familiar the audience is with your topic, or the less hardcore researchy it is, the more time you will want to spend choosing appropriate visuals. Some well-chosen pictures will make your message more sticky. And you might find some images that are so powerful that might make you go back to your script and twist it a bit.

The talk

Once you have the slides more or less ready, you can start preparing the talk itself. Bear this in mind: if you prepare the slides but not the talk, your presentation is likely to suck. Some ideas and tips that can help you in the process:

(1) Tape yourself

As I mentioned before, taping yourself is one of the best tools I have found for improving your presentation skills. It is also an amazing tool for preparing your next talk. If you watch a video of yourself rehearsing the talk, you will be able to analyze what you are explaining wrong, where you are wasting your time, what jokes don't make sense... Besides, it is a perfect timing tool: not only you will get the exact duration of your talk in the video but also how you distributed it. This will allow you to make sure that you are devoting the right amount of time to getting the "main message" through.

(2) Test technical issues as many times as possible

Don't need to mention Murphy's law, I suppose, but if something can go wrong, it will. No matter how much I check things over and over, there are always technical issues that catch me by surprise.

In my last talk, I had a couple of videos that I knew could be problematic. I spent a lot of time in making sure they were working in the presentation. I even tried my laptop with a secondary monitor to make sure. I asked the host well in advance to make sure that I had the possibility of connecting the audio output of my laptop. And on the day, I went to the hall and tested the audio. I was even going to test the videos but the audience was already half in, so I preferred to keep the surprise and tested with some random music instead (big mistake here!). What happened? Videos did not show, so I had to improvise opening them with another program and that did not work very well either because of a limitation on the projector's resolution, I think.

In my case, I like to put myself in situations of technological risks and I like the feeling of doing a complex live demo in a presentation that I know can fail (I guess is like the adrenaline rush I used to have when playing in a concert). But I have to admit that the safest advice is to keep technical challenges as simple as possible. And be anal in checking many times those that you know are likely to fail.

Of course, checking technical details means, among other things, that you need to be in the room for your talk well in advance and test the presentation in the same conditions you are going to it later (even if that means missing one of the coffee breaks in the conference!).

(3) Be ready for improvising

And my last piece of advice may seem to contradict the rest. I have been talking about the importance of preparing many details of the talk. However, a presentation should always leave room for improvisation and adaptation. There is nothing worse than the feeling that the speaker has learned the conference by heart, and is not making any attempt to connect with the audience and the context. Besides, there might be elements during the talk that might force you to improvise: a technical issue, different audience that you expected, a reaction from somebody...

You should be able to put any kind of external element into your presentation while not losing the main message. I don't like having to skip slides since it gives the impression that you are in a rush to finish, but many times there is no alternative: You might have lost precious time in trying to play that video, or maybe went too far in the introduction and now you need to cut short. It is again very important that you have a clear picture of what the "main message" is and improvise by skipping those slides that are not needed to understand it.

I hope that some of this advice is useful in your next presentations. But I would like to hear from you: how do you prepare your talks? Any tips or suggestions you want to share in the comments?

Internship positions on Recommender Systems

2010-09-29T02:52:00.000-07:00

At Telefonica Research we are looking for young and talented researchers willing to expand their horizons by working in an exciting environment in beautiful Barcelona. And, as you will know if you follow this blog, I am particularly interested in working with PhD students whose research focus is Recommender Systems but also neighboring areas such as Data Mining, User Modeling, Social Networks, and Information Retrieval. We offer three month internships and interesting conditions.

Work from previous interns has been published in top conferences such as SIGIR, WWW, Recsys, Web Intelligence... (see my list of recent publications, most of which include interns)

And, if you want references of what an internship in Telefonica is like, you might want to contact some of our previous interns in the group:

Meeyoung Cha, currently Assistant Professor at KAIST
Haewoon Kwak, PhD student at KAIST
Neal Lathia, currently Researcher at UCL
Jae-wook Ahn, PhD student at U. Pittsburgh
Linas Baltrunas, PhD student at U. Bolzano
Denis Parra, PhD student at U. Pittsburgh
Miguel Ramirez, PhD student at U. Pompeu Fabra

Please contact me for more details.

Contextual Movie Recommendations on an iPhone based on Expert Collaborative Filtering

2010-09-23T15:53:00.000-07:00

If you follow this blog, you probably have already read about Expert Collaborative Filtering and The Wisdom of the Few. Maybe you also read about our recent implementation of the approach to recommend music. Well if you are around in Recsys 2010 conference next week you will get to see a demo of yet another prototype on Monday's Demo Session.

We are presenting an iPhone application based on the Expert Collaborative Filtering approach. The application is the result of Josep Bach's undergrad final thesis and you can read the full-blown description of the project in his dissertation. The application, however, is much more than yet another implementation of Expert CF. The main highlights for me is that (a) you can offer personalized recommendations on a phone with 100% privacy guarantees, and (b) you can run a recommendation algorithm on the device, with minimum intervention from the server-side.

Both these issues can be explained by the client-server architecture depicted below. The server is in charge of compiling all the public information available on the web by crawling critic websites like Rottentomatoes. It also gathers information about local cinemas and their schedules. All this information, which again is public, is stored in a SQL database and shared through a RESTful API with devices.

The device, in this case an iPhone but could be anything else, connects to the server and syncs a local database through the RESTful API. Once this is done, all needed information is local on the device. Plus... all the personal information about the user (i.e. ratings on movies in this case). The recommendation algorithm can then run locally and return results in a reasonable time because the set of experts is limited.

Another important addition to the application is that we have added contextual features. The recommendations you will get on the app depend on your location and the time of the day. Therefore, it will recommend things that match your taste according to the expert-based prediction but also are playing in a cinema nearby now.

We haven't done a full user evaluation yet, but informal results are very encouraging. We hope you can come and test it in Recsys and give us your feedback. We will soon post a video. But for now, here are some screenshots of the main app screens.

List of Recommendations given your preferences, critics ratings but also your location

Information on a movie, including critics ratings, your ratings, and also closest cinema that is playing with next show times

Screen showing cinemas near your current location (in blue)

Information on the closest cinema near you

The end of the Age of Search?

2010-09-15T13:26:00.000-07:00

A couple of days ago, there was an interesting article in the New York Times on how social networks are changing the search experience. The truth is that the article is a bit confusing and mixes up several different issues. As a matter of fact, most of the article ends up being an introduction to Hunch, a very interesting recommendation site (thus not "search") based on different technologies including social recommendations.

I twitted the link to the article and presented it as another prove of the "End of the Age of Search". That got me into a very interesting Buzz conversation with Greg Linden on why I thought that the age of search is coming to an end. I promised that I would try to write a more elaborate post to make my point if I had the time... and here I am.

The first time I read about the "end of the age of search" was in an article titled the race to create a smart Google at CNN Money. As a matter of fact, the discussion on how recommender systems were going to render search engines obsolete was cited by Recsys 2009 organizers and turned into their homepage motto.

So, credit given to Jeffrey O'Brien at CNN Money and Recsys09 organizers, I picked up on this idea and have been elaborating it in several of my presentations. Two years ago, for instance, I gave a presentation on Recommendations as the Future of Search in an open research day organized by our lab. The main story, that I have repeated several time since then, is the
following:

"
Think about it: Search is not an ultimate need for people. What people need is information. The fact that they have been using search and this has been so successful is (mainly) because that is the only tool we gave them.

Search by itself is not enough to compensate the ever-growing information overload. First, there is the issue that for most given queries, you will get many more results than a user can ever go through. So you are faced with the problem of how to turn that huge set into the "ten blue links" (i.e. first results page). But, there are more or less smart ways to do so by taking into account context, user preferences and so on.

The main issue, however, is a different one: every search action requires users to explicitly formulate a query. From our geek perspective, we usually forget how difficult it is for a regular user to formulate a query given an information need. Even if you are fairly proficient, it might be complicated to turn a fairly trivial information need into something you can formulate in a simple query (take a look at the experiment I did with some SIGIR attendees when I asked them to search for my daughter's name, which is actually written in my homepage).

So, of course, the bottom line of my story has "traditionally" been that Recommender Systems represent a step forward since (a) they provide ways to assess relevance taking into account personal preferences and context, and (b) they can provide results without the need for explicit queries. I still believe Recommender Systems will have much to say in the way search is handled in the future. Of course, I work on the area, so you might think my opinion is a bit biased. But you don't have to take my word for it. In this year's industry track at the SIGIR conference both Yahoo and Google mentioned "implicit search" as one of the most important trends. Now I hear that Google's Schmidt is talking about autonomous search. They are all different ways of talking of Recommender Systems (which maybe, as my friend @mramirez suggested, is not the sexiest name for a research/technology area).
"

But, we have recently learned some data that can be used as a supporting evidence of the end of the Age of Search. Nielsen just published some results that show a 16% drop in web searches over the last year. And this is something pretty symptomatic! I disagree with some of the comments in that same post by Nielsen saying that this drop is due to the use of mobile devices. Unfortunately, I cannot say that this is due to the huge success of recommender systems either. In my opinion, there is one main reason for this: Search is being replaced by Social Networks.

If you think about it, most information needs users have are not about a particular and concrete piece of information such as "who wrote War and Piece". They are actually much less precise needs such as "what is there to do this weekend" or "is there a cool music album I could listen to while I go to work tomorrow". Or even things like "what important stuff has happened in the world today" or "I need to find a job better than the one I have". If you consider information needs like this, you will realize that the answer is much more likely to come out of your social network than out of an artificially formulated query.

But not only that, I think we would agree that most of the time, when people go into the Internet, they don't have any information need beyond the prototypical "see what's up" or "catch up". And what do they do? They log into Facebook or check Twitter. It is clear that in this cases, search is out of the picture.

Yet another worrying trend for the future of search is the decrease in the use of the web browser and the increase of "walled internet gardens". This idea hit front lines last month with Chris Anderson's piece in Wired and has been covered throughout, so I won't go into it (but this is the main reason I am trying to avoid the use of the word "Web" in favor of "Internet" in this post).

As a finishing note, I don't want anybody to get the wrong impression that by talking about the end of search as the driver for web development, I am implying that search-oriented companies like Google is doomed. Of course not. Google know most of what is in this post as well as I do. As I mentioned, they are more and more talking about implicit or autonomous search as a proxy word for recommender systems. And in the case of social networks taking over search, I am pretty convinced they would agree with such a vision for the future. That is why they have been trying to get into the social scene so hard lately with Buzz, Wave... and more recently the rumors are that they are working in a Facebook killer called Google Me or at least they are looking into adding more and more social features into their search engine.

Interesting times to be doing research in this area because as Search supremacy comes to an end, we will have more space to fill in the void with newer and much cooler ideas.

As always... looking forward to your feedback.

Study on online music taste: call for participation

2010-08-25T01:55:00.000-07:00

Are you a music listener and lastfm user? Are you interested in helping out research while having the chance to win a $600 Amazon gift card? Please help us understand online music tastes by completing a survey that will only take around 15 minutes of your time and might even be fun!

All you need to do to participate is go to this page and provide your last.fm username and a valid email. We will check if you meet the requirements (at least 18 y.o. and 5000 scrobbles on lastfm) and we will then send you a link to your personalized survey.

Thanks for your time!

Multiverse Recommendations (aka using n-dimensional tensor factorization for context-aware collaborative filtering)

2010-08-08T14:29:00.001-07:00

This post is the first of several in which I will be explaining some of the things we are presenting in the upcoming Recsys 2010 conference. The project I will talk about is led by Alexandros Karatzoglou and presents a new approach to context aware recommendations that we have named Multiverse. You can access the full paper here, but I will give you a brief description in this post.

The introduction of context in recommender systems is an area of growing interest. The reason is simple: While we all value the fact that Recommender Systems are able to infer our tastes and recommend new things, it is clear that whatever we like - and are willing to receive - depends on the context. E.g. We do not want to receive the same movie recommendations on TV if we are sitting with the kids on a Sunday afternoon or if we are alone on a late night session. There is a growing body of literature on contextual recommendations. Without going any further, I already posted about context-aware recommendations with micro-profiles on this blog. Also, there is a very good chapter on the topic on the upcoming Recommender Systems Handbook. But, while we wait for it, you might want to look at some of the publications by Adomavicius and Tuzhilin.

Context takes the recommender problem from a two dimensional problem, where we have users and items, to an n-dimensional one where we can have many contextual dimensions added. In our work, we have generalized the successful matrix factorization approach to this n-dimensional case. In order to do this, we have used the idea of tensors, which are precisely a generalization of matrices to n dimensions. The following figure illustrates the idea (note that, for simplicity, we are illustrating the 3 dimensional case with just one contextual variable).

In the paper, we show how this approach outperforms previously existing methods on a number of different datasets. One of these results is illustrated in the figure below. Note how Tensor Factorization (in green) not only outperforms other methods, but it performs better the more contextual information we add. It is also interesting to note how not observing context information (black line) results in worse performance. When we add contextual information to 80% of our data, not using this information yields a result that is almost 50% worse.

The use of context in recommender systems and other areas of information retrieval is a very interesting topic that is likely to get even more attention in the near future. _We will surely contribute to this.

Music Recommendation through Expert-based Collaborative Filtering

2010-07-29T16:07:00.000-07:00

In September I will be presenting the paper entitled "Towards Fully Distributed and Privacy-preserving Recommendations via Expert Collaborative Filtering and RESTful Linked Data" in the 2010 International Conference on Web Intelligence in Toronto. You can read the full paper here, but in this post I will try to give you a taste of what is hidden behind such a long title.

This paper should be understood as a continuation of my research on Expert Based Collaborative Filtering -- the so-called Wisdom of the Few. I recommend you take a look at my previous post on this issue before moving on.

So the basic idea from our previous work was to use domain experts as the only asset for creating neighborhood and predicting item utility in a similar way as is done in standard kNN collaborative filtering. We made some claims of how that method provided many practical advantages over standard approaches. We also claimed that the approach was scalable and flexible enough to be used in many domains. Unfortunately, at that point, we did not have time to implement and prove all that.

The current work presents a practical full-fledged implementation of the approach in the music domain. Our goal is to prove some of the previous claims as well as to stablish an architectural framework for expert collaborative filtering providing, among other things, 100% privacy protection.

The following screenshot will give you an idea of the application. In the client side, it is a Flex/Air stand-alone application that can work in most operating systems. You can rate music albums, see the ratings from the experts, and get personalized recommendations based on that. We also provide access to extended information for albums via links to lastfm as well as access to Linked Data resources from MusicBrainz and others.

The key architectural differences between standard and expert Collaborative Filtering are illustrated in the figure below. Note that in our expert CF, user ratings are kept in the client machine. On the other hand, expert ratings are downloaded into the local machine and the computation for the predictions is performed there avoiding any privacy breach.

The next figure gives some more details of how we implemented the solution in our case. Again, note that the server is only used to crawl and store expert ratings publically available on the web. Those ratings are then queried from the client through a REST-style web api. The computation of neighbors and predictions is then performed in the local machine.

You might be wondering where we got our expert ratings from. If in our previous work, we crawled our movie ratings from rottetomatoes, we now turned to metacritics. The figure below illustrates the number of ratings per critic. In the top positions, we can see AllMusicGuide with over 3500 ratings, or Pitchfork, Uncut, and Mojo, with over 3000.

I believe that expert collaborative filtering is a very flexible and valid paradigm in many domains. It can offer better results than other kinds of recommendations while solving many of the shortcomings such as scalability, privacy, or cold-start. We are currently working in other deployments in the mobile space, for example. But I will explain that in a future post.

Being Social

2010-07-22T09:25:00.000-07:00

This was the title of my talk at the SIGIR Industry track this year. I wanted to post the slides online (see below). However, since there is little explanation in them, I will briefly try to walk you through the story line in this post. Also at the end of the post I added a video of the talk (with some minor gaps), this should also help you get the full picture in case you are missing something.

Being Social

View more presentations from Xavier Amatriain.

The presentation was about some of the projects we are doing at the Telefonica Research Group in Barcelona. In particular, I show some of my projects on Recommender Systems but also others led by Karen Church and Josep M. Pujol

(slides 2-7)
But first let me introduce what is Telefonica for those of you who don't know (probably only applicable if you live in the US or Asia). Telefonica is one of the largest Telecom companies in the world (3rd in market cap). It has had a significant growth in the last 20 years, going from a Spain-only company with 12M customers to operating in more than 25 countries and having over 260M customers. Telefonica I+D (or R&D) is the Research and Development branch. It is the largest private research center in Spain and second largest in Europe. Finally the Research Group of Telefonica I+D has around 20 permanent research scientists covering areas such as multimedia, mobile and ubiquitous computing, social networks, p2p and content distribution, wireless systems, user modeling and data mining, and HCIR.

(slides 8-9)
One of the important issues, not only for users but also for a company like ours, is to find ways to deal with information overload. In very few years we have gone from counting the information we were exposed every day to counting the one we are exposed every second. Twitter streams, facebook updates, photos, videos... It's too much to cope with. Besides, this leads to the so-called "Paradox of Choice", after the very interesting book by Barry Schwartz. Having more choices does not necessarily lead to more freedom. In fact, it often leads to the opposite. If we have many choices, we tend to choose less because of the Analysis Paralysis. And we tend to choose worse, because we oversimplify the choice and use only superficial features.

(slides 10-11)
We tend to think that search engines have the answer to everything, but that is not always true. Actually, searching is not an ultimate human need. Accessing relevant information is. One of the reason search engines are not the ultimate answer to information needs by people is the interface. We technical geeks think that formulating the right query is easy, but this is far from trivial for the non-technical average user.

(slides 12-13)
The good news is that you are not alone. There are many people seeking relevant information. And actually, some of them are your "friends". They might be able to help you find what you need.

(slides 14-15)
I did an interesting test by posting a question on twitter. The question was: "What is my daughter's name?". This information is available in my homepage. Still it is hard to find using any search engine. I received three correct answers hours after. They had all, one way or the other, used my social network (see details of the paths in the slide.

(slides 16-19)
This leads me to the first project, Porqpine, led by Josep M. Pujol. Porqpine is a social and distributed search engine that uses the principle of lazy collaboration by letting users collaborate without extra effort. It allows to find personalized and context-aware answers. And it is stand-alone but can co-exist with other search engines. What is does is to locally cache the page and record user interactions (e.g., bookmarking). Then, searches by querying local caches of a user’s friends. Pages that friends have “interacted with” are ranked higher. It also uses a proxy masking the identity of the friend. It is currently a Firefox addon that can be downloaded here.

(slides 20-21)
We can somewhat overcome content overload by using social input. However, nowadays we are beyond content overload. We also suffer from *context overload*. Our information need also depends on where we are, the time it is, who we are with, what activity we are doing... And this is especially relevant if we consider that the web device of the future is not the desktop but rather the mobile device (be it phone, Ipad...). And a mobile device is not a computer!

(slides 22-29)
Besides of the importance of context, a mobile phone is personal. People also tend to look for more "fresh" content. And there are some queries like "where is the nearest florist?" that are easy to answer. But what about more personal needs like "Where is that cool cocktail bar I went to the other day... I know there were jazz concerts on Thursday and it's near an old church." And what about discovery and serendipity? What about getting help for deciding? Or points of interest in general? or events?

(slides 30-36)
All this lead to a question: Can we improve the search and discovery experience of mobile users by providing a readily available connection to their social? The answer was Karen Church's SSB (Social Search Browser). SSB is an iPhone optimized web-application plus a Facebook app. When launched it centers on the users current physical location and displays all queries/questions posted by other users in that location. As users pan/zoom the set of queries is updated Users can post new queries or interact with queries of others. We did two field studies in Ireland. The surprising results where that SSB became much more than a tool for finding information. It became a tool for helping and sharing experiences and for supporting curiosity. It was actually seen as an extension of people's social network.

So I have shown how you can somewhat minimize content and context overload by tapping onto your social network. Ideally, for most tasks, you want to rely on your close "friends". However, for many information needs, your friends might not be enough and you need to resort to the crowds. We have come to know about the "Wisdom of the Crowds". If I ask enough people, I can be sure that the majority will be right.

(slides 37-39)
But, there is a problem with that: Crowds are not always wise. We don't realize that many times, users are noisy in giving their feedback. Besides, many times our data is too sparse to draw correct conclusions. In our paper "I Like it, I like it not" we studied how consistent people were in giving their opinion. We found very significant inconsistencies especially in mild opinions, but also in negative ones.

(slides 40-44)
So, if we cannot trust the crowds... who can we trust? The experts. As Malcolm Gladwell puts it in Blink, "It is really only experts who can reliably account for their reactions". We know that experts might be biased or trying to steer our opinion. However, even in that case, they will be reliably and consistently doing so. Thus, they can become much better anchor points for predictions. In our "Wisdom of the Few", we presented a Collaborative Filtering approach based on experts from the Web. The basic idea is to find individuals who we can trust to have given reliable opinions on a given domain. These expert opinions are then used to determine who are your most similar experts. The final prediction is then done by computing a standard kNN Collaborative Filtering. Expert Collaborative filtering has many advantages over standard approaches. In particular, it is more scalable and it allows for 100% privacy preservation. This is because, user ratings do not need to be shared on a central repository. Expert opinions can be downloaded locally to perform the computation. We have developed several prototypes including a music recommender system and a mobile cinema recommender with geolocation.

(slide 45)
As a final summary: We all probably knew about Information Overload. But now it is not only that, we also have Context Overload. We can cope with both by using our social network. This means using our friends if possible or the crowds when necessary. However, crowds are not always as wise as they might seem and we are better off using experts.

Hope this is a good enough summary so you get the main message and can follow the pointers to more detailed information. You might also want to watch the talk in the following 3 videos that cover most of it.

Part 1:

Part 2:

Part 3:

Off the beaten track

2010-06-28T15:59:00.000-07:00

Next september, Nava Tintarev will be presenting a paper that she and I co-author in Mobile HCI 2010, in Lisbon. This paper, entitled "Off the Beaten Track - a mobile field study exploring the long tail of mobile tourist recommendations" presents our results on a field study for tourist recommendations. We sent a number of tourists off to visit Barcelona. They were instructed to use a tailored smartphone app which included recommendations of places they could visit.

In the paper, we evaluate the effectiveness, satisfaction and divergence from popularity of a personalized recommender system comparing it to recommending most popular sites. We found that participants visited more of the recommended POIs for lists with popular but non-personalized recommendations. In contrast, the personalized recommendations led participants to visit more POIs overall and visit places "off the beaten track". The level of satisfaction between the two conditions was comparable and high, suggesting that our participants were just as happy with the rarer, "off the beaten track" recommendations and their overall experience. We believe that personalized recommendations set tourists into a discovery mode with an increased chance for serendipitous findings.

This paper is the first of a line of research on tourist recommendations that I have just started and hope to be complementing with new publications soon. I will keep you posted in the blog. In the meantime, you can download the full pdf here.

Temporal diversity in Recommender Systems

2010-06-04T16:14:00.000-07:00

Next month, Neal Lathia will be presenting a paper where I have collaborated in SIGIR. In the paper we address the issues of Temporal Diversity and Novelty in Recommender Systems. You can read the paper here, but I will try to give you a brief summary in this post.

Recommender systems are usually evaluated on their accuracy, that is, their ability to predict how much a user will like/dislike an item given a set of past ratings. However, in any practical scenario, there are many other things that need to be taken into account to evaluate whether a system is giving good an interesting recommendations. One of these relevant issues is the diversity of the top-N recommendation lists. It does not matter that our recommendation is more or less accurate if time after time we recommend the user the same things. A user should expect that the system takes into account her feedback in order to improve and give different and better recommendations.

In the paper, we evaluate the importance of temporal diversity for users through a user survey. Then we analyze the performance of known collaborative filtering algorithms, and we propose different ways to introduce temporal diversity while using traditional recommendation algorithms.

We found several interesting results on how user rating behavior affects temporal diversity. For instance, users with large profiles are likely to see less diversity. However, the amount of ratings introduced since last recommendation correlates directly with more diversity. This suggests that we need to encourage users to rate while implementing mechanisms that prevent profiles from growing too large therefore preventing diversity. Smart mechanisms for rating "aging" might be useful.

Perhaps an even more interesting finding is that, as illustrated in the figure above, different algorithms perform differently regarding temporal diversity. SVD, for instance, is known to be more precise than kNN in the general case. However, it is interesting to note that it is also much less diverse. Therefore, even a simple decision between SVD or kNN as the base of a recommender system cannot be done disregarding issues such as temporal behavior of the algorithms.

Again, much more in the paper and, as always, looking forward to your comments and feedback.

Recsys 2010 Update

2010-05-24T16:03:00.000-07:00

As most of you know, I am co-chairing (together with Marc Torrens) the 2010 ACM Recommender Systems Conference (Recsys 2010 for short) to be held in September in Barcelona. After announcing it in this blog some time back, I thought it was time to give a brief update on the highlights. This is just a summary, but if you want to be up to data, please bookmark the website, or follow us on twitter.

I am listing the highlights in more or less chronological order (or as they come to mind). In no way, the order is meant to imply importance or relevance.

Venue

Hosting a conference in Barcelona is already great. As we explain in the conference website, the city has much to offer. But, what can be better than having said conference in a convention center surrounded by the sea in the harbor, just down from the Ramblas? Well, this is where the Barcelona WTC is located (see picture below). And although we did have other options in the city, we couldn't help but falling in love with the place.

Workshops

Recsys workshops were already a huge success last year in NY. But, judging by the high-quality proposals we have this year, it seems they are still getting better! This year we accepted 8 workshops (actually it is 6 one-day and 1 two-day) and we have accommodated them in two different days: before and after the three days of the main conference. On Sunday Sept. 26th, we have a workshop on Information Fusion, one on Social Recsys, one on Music, and the first part of the two-day Context-aware workshop and challenge. On Thursday Sept. 30th, we have the second part of that workshop plus new workshops on Practical Uses, on e-Learning (in conjunction with the EC-TEL 2010 conference ), and one on Recsys evaluation.

There is so much to choose from that the problem will be trying to decide which one not to attend. I am really looking forward to all of these workshops.

Tutorials

We have also line-up three very interesting tutorials by key figures in the field.

Guy Shani, from Ben Gurion University, will be giving a very much anticipated tutorial on Evaluating Recsys.

Joe Konstan, one of the fathers of the field and chair of SIGCHI for several years will be introducing the use of HCI techniques in Recsys.

Finally, Ricardo Baeza-Yates, VP of Yahoo Research, will be talking about predicting and recommending queries. Apart from being in my PhD committee, Ricardo is the only researcher I know who has a publication with more than 7000 citations.

Papers

We still don't know many details about the main attraction of the conference: the accepted papers. However, we do know that the submissions went up from last year. Bearing those numbers in mind the anticipated acceptance rate will be below 20%... which takes the conference to levels of 1st tier.

Hotels

We have arranged a great list of hotels in Barcelona so you have plenty to choose from. If you can afford to stay in the Eurostars Grand Marina, that should be your first pick since it is located right on the conference venue and it is an amazing hotel. However, we know most budgets are tight nowadays so we have included two great 4* hotels (conveniently located and both top 50 out of 600 in tripadvisor). We have even added a more affordable 3* hotel that is also conveniently located and has good reviews.

To be honest, if I had to travel to Barcelona myself, I couldn't find better choices than these.

Local Festivity

Each year, Barcelona celebrates its major festivity by the End of September. This year, major events will be scheduled 23-26th, just before the start of the conference. If you want an even better taste of our local culture, we recommend you come a couple of days earlier and enjoy the festivity. Visit the Merce website for more details.

All in all, we are looking forward to a great conference and hope to see you here!

How many rejections make an acceptance?

2010-05-03T13:37:00.001-07:00

Now that I have a decent (though not impressive) publication record, I realize that people might think that it was easy to get here. Not only that, some of you now starting with your first rejections, might think that researchers like me were fortunate enough not to get the kind of reviews you are getting. Even worse, you might think that we are better than you. Here is a secret for you: I have got many more rejections than acceptances during my career. And, although I have managed to improve my stats significantly (experience counts, that's a fact), I still do!

I don't want to generalize here. I am sure there are researchers that manage to get all the papers in, starting on their very first submission, and only rarely get pissed at a rejection. But to be honest, I don't know any of them. Top-tier conferences and journals are designed to have low acceptance rates (10 to 20%). Also, they are supposedly designed not to let "bad stuff" in, but they are not optimized to make sure "good stuff" is not left out!

Here is the best kept secret for researchers: you have to learn to digest your rejections and endure.

A case study

As I was writing my last post on the paper I have published IEEE Transactions on Software Engineering, I was thinking on the long story of this particular submission and thought it might be good to share it. My goal is not to complain about how the publication system works (although there is meat there for several posts), but rather to encourage you to persist and not give up. I have omitted names of journals to preserve anonymity and to avoid pointing the finger.

My first move was to send the paper to a 2nd-tier journal. I was working on other journals at that time and this one was not a priority. I simply wanted to get the word out of that part of my thesis. This journal seemed ideal. The result is that, for the only time in my life, I got the paper rejected without revision! Yes, no reviewers or anything like that. Just the editor telling me that the paper was not relevant for the journal. As later events proved, this was pretty much bullshit.
Because of the disappointment and the work I had put into it, I decided to send it to another journal. Only that I decided that would be a top 1st tier journal instead. This time, I did get reviews. I have to admit that the reviews were insightful this time. The main issue was that the reviewers wanted me to focus more on the practical case and less on the general model. So I got a Major Revision out of this. I am pretty sure I could have gone for it at that point but decided not to. Again, I had other papers to focus on and this one was not my priority. But I kept the paper and the reviews.
The story of the paper would have ended there, if it wasn't because I received an invitation from a researcher I know. He was guess editor to a special edition on the same journal I had sent on (2) and, because he knew my work, was inviting me to submit. Why not? I pulled the draft out of the drawer (figuratively, that is) and started working on the reviews I had previously received in that same journal. Right, isn't it? No, wrong! It turned out that the reviewers were complaining about completely different things. Because this was a special issue, all reviewers were uber-experts in this field. And they questioned technicalities and semantics as if they were very importance. However, the reviews were not bad at all. Actually, the paper might had made it this time to the regular journal. But because it was a special issue and they could only accept 5-6 papers out of 50 submissions, it was rejected... again!
So here I am, with a paper I had started working on 5 years before and I knew had not been accepted for bad luck. I took it more or less like it was and sent it to an even better journal (as a matter of fact, out of revenge, I just looked for the journal on software engineering with the highest impact factor). And guess what, it was accepted! I did have to do some revisions though. The funny thing is that those revisions were essentially meant to take the paper to how it was in step one since reviewers asked for less details on the practical case and more meat on the general model.

End of the story. So I have had a paper accepted to a top journal like IEEE Transactions on Software Engineering. This paper is essentially the same that I had 5 years ago and was rejected without review by a crappy second-tier one. And in between, I had to steer one way and the other several times... what a ride!

Y. I am convinced! I will send all my drafts tomorrow and hope they eventually make it...

NO!

I don't want the post to be understood as a call to send crappy research anywhere you can in the hope that it makes it at some point. This doesn't work. Period. I review enough papers to know that people send unacceptable stuff (specially to journals). I am talking about papers that are clear reject after you have spent 5 minutes reading them. I feel the pain, and I am the first one that does not want to waste my time reading pseudo-papers that are sent in the hope that nobody notices how bad they are.

Y. You are so right there! People who send unacceptable things for review should be banned from the community, their paper published on a website for everybody to see, and...

Wait, hold on a sec. That is not the way to go either!

Maybe you are reading this post and were fortunate enough to do your PhD in a top US university. And you had an advisor who called you into the office twice a week. An advisor who read each of your drafts 20 times and marked it all in red for you to repeat many times. If you are in this kind of situation, I have some news for you: you are the exception, no the rule. Many PhD students go to just decent universities and are lucky if they have an advisor that is remotely interested in their thesis. Even in that case, it is likely that the advisor has too many things to do to provide any kind of regular feedback.

Students and junior researchers use submissions to get feedback on their research... and to learn. And you know what, I think that, as much as I hate reviewing bad papers, they are right to do so. What do you think?

You're not alone

To finish this post/rant, I wanted to point out that having great ideas rejected is part of human history, you are not alone. Read, for instance, this list where X. Sala i Martin has compiled famous ideas that were rejected (some of them, like the xerox machine, time after time). Also, this post by Michael Nielsen has a very interesting discussion on the reliability of peer review, where he cites other examples of famous scientific ideas that were rejected.

As always, your comments are most welcomed!

Frameworks generate Domain Specific Languages

2010-04-30T16:22:00.001-07:00

"Frameworks generate Domain Specific Languages: a case-study in the Multimedia Domain" is the title of a paper I have authored and has just been published in IEEE Transactions on Software Engineering. As you might have guessed, the title of the paper is a twist on the classical "Patterns Generate Architectures" by Kent Beck and Ralph Johnson.

This publication completes the trilogy of journal articles I intended to get out of my Thesis (two others are "A Domain-Specific Metamodel for Multimedia Processing Systems" in IEEE Transactions on Multimedia, and "A framework for efficient and rapid development of cross-platform audio applications" in ACM Multimedia Systems). My intention was to get this paper published a long time ago but I will save that juicy story for a future post.

But, in any case, what are we presenting in the paper? Before explaining this, you might need to understand, in case you don't already, what a domain-specific language is:

A Domain-Specific Language (DSL) is a high-level programming or modeling language tailored for a specific domain. A DSL lacks the flexibility of general-purpose languages but it offers powerful constructs that are closer to the concerns and constructs of the particular domain.

In this work we claim that the aim of any software development process should be to come up with a DSL. In order to do this we bridge the gap between Model-driven Architectures (MDA), Pattern Languages and Framework Development and embed all into a coherent and agile development model. This is illustrated in the figure below:

Here is the transcript of the abstract (complete preprint pdf here):

"We present an approach to software framework development that includes the generation of domain-specific languages (DSL) and pattern languages as goals for the process. Our model is made of three workflows -- framework, metamodel, and patterns -- and three phases -- inception, construction, and formalization. The main conclusion is that when developing a framework we can produce with minimal overhead -- almost as a side-effect -- a metamodel with an associated DSL, and a pattern language. Both outputs will not only help the framework evolve in the right direction but will also be valuable in themselves.

In order to illustrate these ideas, we present a case-study in the multimedia domain. For several years we have been developing a multimedia framework. The process has produced a full-fledged domain-specific metamodel for the multimedia domain, with an associated DSL, and a pattern language. "

The Hadoop Ecosystem (a personal overview from a non-expert)

2010-02-28T15:26:00.000-08:00

A couple of weeks ago, I attended a three-day course on Hadoop from the guys at Cloudera. Although I had heard and read about Hadoop before, this was a great opportunity to learn many details on Hadoop and find out about several tools that make up the Hadoop ecosystem. If, like me before, you only have a rough idea of what's in Hadoop, you should be interested in the post. Take what I say with a grain of salt, since I am no expert in Hadoop. However, because I am not an expert, I think I can guarantee a fresher look and you can trust I am not trying to sell you the project. But, if you are an expert and you read the post, you might want to give feedback in case I got something wrong.

Hadoop is an open source java implementation of the MapReduce framework introduced by Google. The main developer and contributor to Hadoop, however, is Yahoo. It might seem weird that one of Google's main competitors releases an open source version of a framework they introduced. More so, when Google has recently been granted a patent for it. However, it seems unlikely that Google can execute their patent. One of the main reasons is that Map and Reduce functions have been known and used in functional programming for many years. Another reason is that Hadoop has gained a huge popularity as part of the Apache project. Enforcing the patent would not get Google much love from many companies that are now making a living of it or using it as an important component in their web architecture.

But, before we go into any more detail, it would be good to understand what can Hadoop be used for and when we should think about adopting it. First, and above all, Hadoop is a framework for data analysis and processing. Therefore, if you have no data, or if you have no need to process it, do not continue with this post. Hadoop is sometimes presented as an alternative to traditional relational databases. However, it is not a database (although it does provide a noSQL one called HBase as one of its tools), it is a framework for distributing data processes. Ok, so here was the second keyword: distribution. If you think you can do whatever you need to do in a single machine, you don't need Hadoop. However, you might want to look at it anyway, since distributing your data processes can be cheaper and also much more reliable. And finally, and related to the previous, using Hadoop only makes sense if you are processing large datasets and by large I mean several TB's.

However, even if your problem fits into the three previous conditions (distributed processing of large datasets) you can still not be completely sure Hadoop is your solution. Distributed relational databases are still an option. I won't go into the details, but you might want to read at some voices that are recently stepping in to defend the scalability of relational databases and their applicability in highly demanding large datasets. These two posts are good reading: "Gett ing Real about NoSQL and the SQL-Isn't-Scalable Lie" and "SCALE 8x: Relational vs. non-relational". I would also recommend this recent presentation on "What every developer should know about database scalability"

So now that we have some intuition of when Hadoop may be of interest, let me introduce the two main issues behind Hadoop: MapReduce and HDFS.

MapReduce is a programming model introduced by Google, which is at the core of Hadoop. It is based on the use of two functions taken from functional programming: Map and Reduce. Map processes a (key,value) pair into a list of intermediate (key,value) pairs. Reduce takes an intermediate key and the set of values for that key. Both the mapper and reducer functions are written by the user. The framework groups together intermediate values associated with the same key in order to pass them to the corresponding Reduce.

MapReduce claims to be a sufficiently generic programming model that most data processing tasks can be decomposed in such a way. If you are interested in learning more, I recommend you start with Google's paper. You can also take a look at Google's set of videos introducing the framework. If you want a more "academic" presentation, you might want to take a look at these UC Berkeley classes.

The other important core issue in Hadoop I mentioned before is the Hadoop Distributed File System (HDFS). HDFS is the equivalent of the Google File System (GFS) used in the original MapReduce framework. This filesystem is optimized for reading in streaming large files (from several gigabytes to terabytes). Note that HDFS does not allow, for instance, to edit a file once it has been written.

Ok, so now we have the basics in place: how do we use Hadoop? Since Hadoop is written in Java, the most straightforward to get started is by using its Java API. If you look at the Hadoop Map/Reduce tutorial, for instance, you will see how the framework is introduced through its Java API.

But, if you want to use Hadoop but would rather keep away from Java, there are plenty of other options. First, there is Hadoop Streaming, which allows to use arbitrary program code with Hadoop. Stdin and Stdout are used for data flow, and each mapper and reducer is defined in a separate program. This comes in very handy if you want to use Hadoop through a scripting language. Now, if you want to have a greater performance in your mapper and reducer functions and would like to call compiled C++ code instead, your solution is called Hadoop Pipes.

Now, what if you still would like to access the Hadoop framework but do not fancy the MapReduce programming mode? In other words, is there any higher-level and more programmer friendly way to interface with Hadoop? And the answer is, of course, yes. There are several ways to do this but I will mention two of them: Hive and Pig. Hive is a tool developed at Facebook that allows for an SQL-like access to the Hadoop infrastructure. Although the project is not very mature yet, this is a very interesting option to consider and it seems to be giving very good results to Facebook. The other option is to use Pig, developed by Yahoo. Pig provides a higher-level language called Pig Latin that increases productivity, especially if you are dealing with non-java programmers that are closer to the domain (e.g. data analysts). Pig Latin is a dataflow language and it even has a graphical front-end plugin for Eclipse called PigPen.

I would not like to finish this personal overview of the Hadoop Ecosystem without mentioning
Mahout, a project for distributed machine learning with Hadoop. Among its examples, Mahout includes an implementation of several collaborative filtering algorithms for recommendation. I would also encourage you to take a look at this list of academic papers about or using Hadoop.

At this point I have to say that I have mixed feelings about Hadoop and about MapReduce itself. Although it is a powerful framework with immediate application to real-life problems that involve very large datasets, the model feels more like a kludge than a paradigm shift. I understand why people turn to tools like Hive and Pig that hide the MapReduce complexity behind more friendly models such as ER and Dataflow networks. Providing a framework that is both efficient, usable but also conceptually illuminating is definitely an area to work in the future. And it seems that I am not the only one thinking along these lines. Even Yahoo themselves are looking into new ways that go beyond Hadoop and MapReduce.

Is Scrum really (that) evil?

2010-02-02T15:39:00.000-08:00

Some days ago I was surprised to find (the cult of) Scrum in a list of the Worst Technologies of the Decade. I posted the link on LinkedIn's Agile Alliance group and that started a mild discussion where most people more or less defended Scrum. What follows is my personal take on the issue: Is Scrum really (that) evil?

(No need to say that if you know nothing about Scrum you should at least learn something about it before proceeding with the rest of the post)

Ok, so the first thing people will tell you about Scrum is that it is not a method, obviously not a technology, not a methodology... then what is it? Scrum sells itself as a ... (drums here) "framework"! For those of you with a less agile background, another well known example of a framework in the context of development methods is the Rational Unified Process. If you are thinking on proposing a new development "method", you should definitely think about selling it as a framework. This will bring you a number of benefits, the most important being:

You can say that your framework is useful for any kind of situation
If anything goes wrong, you can always blame whoever instantiated the framework for not doing it right

Well, that is exactly what happens with Scrum. You can not say it is evil in itself, not even good or bad: it will be as good or bad as its particular instance. This is, of course, unfair with Scrum users that may have the feeling that if anything goes right it will be thanks to the framework, but if anything is wrong it will be their fault.

(UPDATE: The following paragraph has been edited after feedback from comments to the post and Kent Beck himself. I only had time to skip through the 2nd Edition of XP Explained, but I think I got the point.)

Unfortunately, the alternative is not very appealing either. Other methods like eXtreme Programming in their first incarnations would tell you that the only way to apply the method was to apply all practices. There was an important point in the message since some practices are actually complementary and feedback into others. So you can't, for instance, decide to skip Continuous Integration and still think you can get away with it. In any case, reality is that you cannot expect all teams to apply all practices and while practices like Unit Testing reach 60% usage, others like Pair Programming do not even reach 25%. Also as Kent Beck acknowledges in the preface to the 2nd edition of his foundational eXtreme Programming Explained, enforcing all practices is like enforcing a given programming style. And that might not be fit in every situation. The current proposed solution in XP is to present some primary practices and some corollary. Also now the message is that practices cannot be enforced and need to be evaluated in each situation. This brings XP closer to being a toolkit of practices, which is not far from the concept of framework: the responsibility is on the particular instance or application.

Maybe the ideal solution would look something like: propose a generic framework that includes several practices and variations and then illustrate the framework with several practical instances that can be used depending on the project at hand. Of course, I am no the first one to think about something like this. As a matter of fact, Alistair Cockburn's Crystal methodologies were designed precisely with this idea in mind. Unfortunately the author did not get very far in proposing the different instances and now the only one fully developer is Crystal Clear, valid for smaller and lighter projects.

In any case, Scrum is a framework and when something goes wrong it is pretty likely that the "user" will be to blame. However, as I have just explained, there is no clear alternative to putting so much weigh on the particular instance of a method.

Still, can we say that there some things that are inherently good or bad about scrum?

The good things

Of course there are a few things that are good about Scrum regardless of how you apply the framework. To name a few:

Accountability: Scrum makes it much easier to make teams accountable for what they do. It also makes it easier to hold clients accountable for what they asked and even managers for what they did not provide. Forget about eXtreme Programming cards and boards, scrum artifacts like the burndown charts or the impediments or product backlog are great tools if what you need is more accountability in your life... I mean projects.
Credibility of Agile: Let's face it, the first thing that comes to mind to many people when they read the agile manifesto is a bunch of long-bearded hipster coding with one hand and eating chips with the other. And this impression might not improve much more when you mention things like the "planning game" or.... (my god!) Pair Programming. Scrum brings credibility to Agile because it looks like "serious stuff".
The agile for non-developers and managers: Believe it or not, those who have to decide whether to promote Agile in a company may have no clue of what the life of a developer is like (and many could not care less). If you plan selling Agile by saying you will make developer's life easier, think again. But if you talk about shortening cycles, minimizing risk and... improving accountability (that is offering tools for managers to understand what is going on) you might have a chance. The biggest strength of Scrum is that it makes it much easier for managers and decision makers to buy into Agile. And believe me, this is a major issue that by itself may justify Scrum.

The bad ones

Too many artifacts: Ok, but wasn't one of the goals of becoming agile to offer a lightweight method with as few artifacts as possible. Then, what are all these artifacts doing in my life now? Product backlog, sprint backlog, impediments backlog, product burndown chart, sprint burndown chart, daily meeting, retrospective, sprint planning... is this really necessary? The truth is that if managed right many of these artifacts are not a huge overhead and are useful, really. And again, they are needed to convince managers that we are doing things right. The problem is that, if we are not very careful, it is very easy to abuse them and end up making them more important than the product or team itself.
Too much focus on the process: Related to the previous, it is easy to fall into a situation where the process is more important than anything else. It is not about your project, your product, and your team... it is about getting the process right. What? This sounds more like CMM than Agile!
Developers? What developers?: And finally... where does that leave developers? If you implement Scrum and nothing else, you are pretty likely to turn developers' lifes into a sweatshop 2.0: they have to work more, faster, and better.... and are more accountable than before. Scrum does not care much about developers, so you better do this yourself. That is why a preferred approach is to sparkle in some XP practices on top of a Scrum-driven project organization. However, because XP will not be so popular among managers and decision makers you are likely to have much less support (if you don't agree with this, just try mentioning pair programming). Scrum by itself is not developer-friendly and that is a fact.

Conclusions: is this really agile?

So now do me a favor and take a moment to (re)-read the Agile Manifesto. What do you read in the first line? "(We value) Individuals and interactions over processes and tools". The main sin is that it makes it really easy to revert the sign in the equation. It is much easier to sell processes and tools to managers and because Scrum is a framework it is very easy to in a way that is not even agile anymore. of Scrum

My advice, however is not to avoid Scrum. Use Scrum to convince your organization of the benefits of Agile. Use the "processes and tools" view to make your point to managers and decision makers. But,

Do not forget to build motivated teams
Value developers and educate them in eXtreme Programming techniques
Make them feel that they are far more important than any Scrum process or tool by allowing self-organization
Make sure that Scrum Masters (or project managers and the like) understand they dual role of using the "process" to improve accountability and upwards reporting while using people's skill and valuing the individuals in the team.

Of course, this is not the magic bullet of agile implementation. As a matter of fact, the combination of Scrum with eXtreme Programming techniques is already a favored approach in many companies (according to the 2009 State of Agile study 24% of the companies use this hybrid). But again, individuals and interactions may be more valuable than any process or tool. So at the end, it may not be so much about how you combine or instantiate your particular flavor of agile method but about how good and motivated are your teams.

As always, would love to hear your comments and experiences.

The hidden complexity of survey design (Part 1)

2009-12-20T14:42:00.000-08:00

A couple of weeks ago I attended a two-day course on Survey Design and Evaluation. In my recent research (see for instance the Rate It Again publication in last Recsys conference) I have become more and more interested on how people give their opinions.

The course was taught entirely by Professor Willem Saris, a very well-known researcher in survey design that was able to attract attendees from all over the world for this course. Although the course was fairly advanced, it touched upon the very issues that I wanted to see discussed. In this post I will try to very briefly mention some of them. More than trying to give a thorough explanation, I hope to draw your attention over some of these issues. Even if you are not into Recommender Systems, it is not strange for Computer Science researchers to be involved in projects in which you need to do some sort of survey, and I am sure you will find some of these issues as interesting as I have.

I will summarize some initial issues in this first post and dive into others in future posts if you consider it interesting enough.

But, before I start, let me throw in a couple of surprising conclusions just to catch your attention:

Batteries of agree/disagree questions are evil! Yes, I am sure you have come across them and possibly even designed a survey in which users are asked at the beginning something like "Mark how much you agree/disagree with the following statements". Well, this is to be avoided at any cost. I will clarify why and how you can replace these kinds of questions.
You cannot compare results among different demographic groups assuming that the same response means the same to any group. It turns out that different countries, for instance, have different rating styles and understand questions differently. For instance, British respondents tend to be much milder in their response than Spaniards. In a scale from 0 to 5, a British 3 might mean the same as a Spanish 5! In any case, this is not something you can assume in advanced, it is something you need to analyze in order to guarantee fair comparisons. More on this later.

Ok, so I hope I have caught your attention by now and you agree with me that these issues are very interesting and seldom explained (actually, during the course we saw many examples of professional surveys that were plain wrong).

The method for developing a survey presented in the course was a three step procedure: (1) Distinguish between concepts by postulation and concepts by intuition; (2) Develop assertions for concepts by intuition; and (3) Develop requests for an answer from assertions. Let's see them in a bit of detail.

1. Concepts by postulation and concepts by intuition

One first important decision when trying to measure a given concept is whether we can measure it by intuition or by postulation. If we think that a concept is straightforward enough, we can directly ask the question we would like to be answered (e.g. How often do you watch sports on television?). However, many times we are trying to measure concepts for which a simple and direct question won't do (e.g. How interested in politics are you?) so we need to measure them by postulation.

When measuring a concept by postulation, we need to decompose the complex concept we want to measure into a series of indicators. These indicators can be either formative or reflective. Formative indicators are variables that define the concept. They should take into account all the necessary components and are not necessarily correlated. On the other hand, reflective indicators are consequences of the concept being measured (e.g. people watch the news because they are interested on politics). These indicators are correlated since they are all linked by the originating concept.

2. From concept to assertion

There are three forms of assertions for asking a concept by intuition:

Subject + LV predicator + subject complement (e.g. Politicians are fair)
Subject + predicator + direct object (e.g. I like conservatives)
Subject + predicator (e.g. The importance of world economics has changed)

On the other hand, a concept by intuition might be measuring different kinds of subjective variables that can be separated into categories such as: evaluation, importance, feelings, rights, policies... It turns out that depending on the kind of subjective variable we are measure, one kind of structure might or might not be appropriate. For instance, if you are measuring importance, only structure 1 will work (there is a complete table that I cannot reproduce where you see the relation between kind of variable and structure to use).

3. From assertions to requests for answers

One last step is to decide how to present the request to the survey participant. The following list summarizes the different options available:

Direct request

With WH word
Without WH word

Direct Instruction ("Please indicate....")
Direct Request ("Will you vote...")

Indirect Request (made of pre request and subordinate clause)

With WH word ("Tell me why you think...")
Without WH word ("Do you think....?")

Following this 3 step approach does not guarantee you are avoiding all errors but rather guarantees that you are looking into all the issues that are needed in order to decide what is the right question to measure a given concept.

And, if you cannot avoid all errors, what can you do about it? Well, you can measure them and take them into account and even predict them. In order to do that I would need to introduce the Multitrait Multimethod Approach and concepts such as reliability, validity, and quality. But that shall be in a second post if there is enough interest on this.

You can read more on these issues in Wille Saris' book "Design, Evaluation, and Analysis of Questionnaires for Survey Research".

On the uselessness of content for recommendations

2009-12-13T06:44:00.000-08:00

This is one of the hot discussions that has sparked as a result of the Netflix Prize. During the competition several teams reported trying to use movie metadata always with discouraging results. This is probably best summarized by a 2008 post by Pragmatic Theory, one of the leading teams.

The issue was re-opened during the last Recsys conference in two ways: First, there was an interesting discussion during one of the panels including the leading teams. Second a paper with a rather provocative title was published: "Recommending new movies: even a few ratings are more valuable than metadata" .

After this, I have seen several discussions in which people used these findings to conclude that content-based recommendations are little more than a dead end, and it is not worth to invest on such research. One such discussion happened in the Recommender Systems group in LinkedIn. But, it was in the Music-IR list, where things heated up the most, turning into a long and interesting thread. Most of what follows is basically an edited version of what I already expressed in those two discussions.

In a few words, my take on this issue is that results reported in the context of the Netflix competition are (1) Algorithm-dependent and (2) dataset-dependent. Although these findings are a valid explanation of why people found no use for metadata in the context of the Netflix prize, one can not extrapolate this finding to other contexts. Why?

Results related to the Netflix prize only refer to how some specific content features help improve the success measure chosen in this case (RMSE). It is a well-known fact that RMSE in a Recommender System does not correlate perfectly with user satisfaction. Things like, for instance, serendipity or novelty, are more likely to come out of a content-based than a CF Recsys since content-based approaches are better suited to explore the long tail.
The dataset in Netflix is somewhat representative of many Recsys cases, but not all. For instance, the sparsity of the rating matrix is much greater in the "movie" dimension, than in the "user" dimension. That is, for a given movie, we are likely to have many ratings. On the other hand, for a given user, we are likely to have very few ratings. As some of the participants in the Recsys panel explained, the Netflix problem is more about how to fill in user "missing values" than movie "missing values". That is one of the reasons why movie content does not help much. Adding content to the user dimension (for instance by adding demographics) would probably have helped. Obviously, this is not easy to do unless Netflix had included the phone number or SSN of users in the dataset.
When people talk about content information in the context of the Netflix Prize, they are referring to a very specific form of content: editorial metadata coming mainly from imdb. But, in different settings, there are many other and better sources of content information. For instance, one can try to infer descriptors by automatically analyzi ng the signal (either video or audio) and use those features for content-based recommendations. We are still far from having automatic algorithms that can on their own bring useful enough features to map to user preferences. But, that does not mean these features do not exist. Another approach to extracting those features is to have experts manually anotate the content. This is what Pandora does in their music recommendation system. And although I have not seen hard numbers, it seems users are more satisfied than when using CF alone.

I think that we will probably see the use of content (and user demographics) in the second edition of the prize, since the dataset will be very different and will include fewer ratings per movie and more user info.

All that said, in the general case, and with no other info on the problem, I would probably venture to say that Collaborative Filtering is a more general solution than content-based. But clearly, the best solution is to combine both as each solves a part of the problem.

So, let me try to summarize my thinking in a set of simple statements:

CF is more effective than content-based recommendations in the general case.
The fact that editorial metadata has not proved useful to increase RMSE accuracy in the Netflix Prize does not mean that content-based recommendations are useless.
Adding some sort of content description helps recommendations as long as this description does effectively describe the content and maps into user preferences.
Editorial metadata does not map directly to the content, neither to user preferences so its usefulness may be very limmited.
Feautures automatically derived from the content map directly to the content but not to user preferences in the general case. Lots of research efforts still need to go into this to close this semantic gap.
Manually annotated content features map to the content and to user preferences so they should prove useful as in the case of Pandora. But they might be expensive in the general case.

As always, looking forward to your comments.

Recsys 09

2009-10-26T01:52:00.000-07:00

Last week I attended the 2009 ACM Conference on Recommender Systems, Recsys09 for short. The conference took place in New York University's Stern School of Business organized by Alex Tuzhilin. This was the 3rd edition of this very special conference for me. Special for several reasons such as the fact that it is the main conference in the area that I am focusing my research; or the fact that I am co-chairing the conference next year in Barcelona. The area of recommender systems has also a special attraction since it combines people with backgrounds as different as HCI, Marketing, Data Mining, Information Retrieval, or Mathematics. If you add the fact that there is an extremely important representation from industry, and many of which you won't easily see in many other conferences from Netflix to Autodesk and a great number of start-ups, you have an explosive cocktail. People in the audience that rave when they see a formula that cannot fit into one slide mix with senior committee members that propose to automatically reject papers that use the Greek alphabet.

The conference has been steadily growing for the past years. It started out of a workshop organized in Bilbao by Strands. The fi rst edition was then held in Minneapolis, home to the Movielens group which could also be considered birth place of the area as a whole. Then off to EPFL and finally this year in NY. The numbers are astonishing for a conference as young (and presumably focused) as this one: more than 280 attendees and an acceptance rate of 19% make it look almost like a first-tier conference.

If you want to get a good idea of what went on during the conference I recommend you take a look at the tweets hashed with #recsys09. And if you want a really quick idea of what where the core topics, look at the beautiful tag cloud below, generated from the tweets by Barry Smyth. In the next paragraphs I will briefly highlight what I think were the most important ideas discussed during the conference.

The first day, we had 3 very interesting tutorials. These tutorials had the great virtue of already setting what would be 3 of the most important topics during the conference: Social Recommendations and Trust, Algorithms, and the Netflix Prize.

In the first tutorial, Jennifer Golbeck did an awesome job of introducing the field of Trust-based Recommendations and explain the challenges in the field. The tutorial was extremely interactive with many questions and comments from the audience. It is true that the idea of trust is also one that very easily leads to passionate debates and opinions. The area of trust and social-based recommendations appeared again and again during the conference. There was a whole session devoted to it in the main track (or 2 if we include the one on tags and Social Networks) and a workshop on the last day. Interestingly enough, though, I did hear relevant people from the industry say that they did not believe social recommendations to be of any practical use. Don't really know what to make of that though.

The second tutorial was more of a traditional and classical lecture on Bayesian Methods. Bayesian Methods is the most popular (but not only) approach to model-based recommendations. They have two main advantages: they allow for the use of nice probabilistic formalisms, and they allow to infer knowledge from the resulting model. However, latent models based on Matrix Factorization have proved to be more reliable and, in principle, they also allow to infer knowledge from the latent variables. During the conference there were 2 different sessions on algorithms, which were dominated by different approaches to hybridize recommendations and by improvements over pre-existing collaborative filtering methods. Among the latter, I should mention the Best Paper winner, Benjamin Marlin. His paper proves that missing data (i.e. items that have not been rated) cannot be considered random and he introduces a way of taking some non-random effects into account. I found the conclusions of the paper not very striking, but the approach and scope of the idea is. And Marlin deserves the award for being the first to point to this issue, and also for all his great work in the area in general.

The last tutorial in day 1, which started a thread of its own, was a discussion on the lessons learned from the Netflix Prize. Very, very interesting discussion where some of the issues I mentioned in my previous blog post were brought up. For instance, I asked about the goodness of RMSE as a success measure. Everybody agrees that the only way to really evaluate a recommender is to do A/B tests on a real system but you cannot do this in an unsupervised way such as the contest. However, I insisted on the possibility of using other measures such as top-N related ones (e.g. nDCG). The (not very convincing) answer to this possibility was from the participants: it would be much harder to optmize algorithms for top-N measures that for the much more simple RMSE. The Netflix prize appeared now and again during the conference, especially since it was finally awarded recently. For instance, there was a very provocative paper by one of the participant teams proving that metadata is useless. This has stirred a heated discussion on whether that means that content-based approaches are useless altogether. The simple answer: NO. They are useless in the very specific case of the Netflix competition and dataset, and using RMSE as the success measure. Content-based approaches (and hybrids) are here to stay and need much more research.

The last thread that was also started on the very first day was the industrial one. As I mentioned before, company presence in Recsys is very relevant. And this year it was kicked of by a panel where Netflix and Yahoo discussed on the 8 challenges of the Recommender Systems Field. The panel was extremely interesting because John Riedl did a great jog on conducting it and on getting the two industry particpants to prepare it for weeks. To summarize, the Challenges were: transparency, exploration, navigation, time value, user action interpretation, evaluation, scalability, and relation academy/industry. The next industrial activity in the program was Francisco Marin's keynote where instead of the challenges he talked about the 10 lessons learned during his years of experience. It was a brilliant keynote that impacted many people (especially some students that were then deciding to change the orientation of their PhD). In Francisco's vision the algorithm is only 5% of the Recommender, while the most important part is the User Interface, which should take around 50% of the resources. But, if you want an excellent summary of this keynote, take a look at Neal Lathia's reconstruction from tweets. The last activity worth mentioning from this industrial thread was the Industry Workshop on the last day. It was organized by Marc Torrens (the other co-chair of next year's conference) and it attracted more than 45 people from industry.

A final thread that did not start on the first day was the application-related one. There was an applications session that was a sort of miscellaneous but where Jill Freyne presented a very interesting and well-delivered paper on the effect of people recommendation on social networks. In this application thread I should include some of the very interesting posters in the poster session. Applications that went all the way from a source code recommender from Karatzoglou and Weimer to IPTV or mobile tourist recommender systems.

Anoother very interesting thing left out of these 5 thread was the Workshop on Context-aware Recommender Systems where I presented some of our preliminary work on time-dependent music recommendation.

As a final personal promotion note I should say that my paper was probably an interesting oddball in the conference. It was the only paper that addressed the issue of data quality and user feedback and the impact it has on the recommendations. It made it really tough on the organizers to decide what session it should belong to, so I ended up presenting in the Trust session. But my impression was the it was very well received and i opens up a whole new avenue of future research in the field. Here you can check the slides I used during the presentation.

Rate it Again

View more documents from Xavier Amatriain.

Overall, a great conference. And although the bar was set very, very high, we hope to exceed expectations in our 2010 Recsys in Barcelona. Hope to see everyone there!

(Btw, this is a very personal overview. Feel free to leave you in the form of comments and let me know if there is any mistake or misinterpretation)

The Netflix Prize: Lessons Learned

2009-09-29T15:19:00.001-07:00

Some time ago I published a post with the title "What if there is no Million $" in which I discussed the possibility that the Netflix prize had no solution. A few weeks later, two teams beat the 10% threshold that entitled them to the grand prize. Bellkor's Pragmatic Chaos beat The Ensemble in a photo finish, only because they sent their solution 20 minutes earlier.

If you want more details on how it all happened I recommend you start by reading the two winning teams web pages. The Ensemble (runner up) has an awesome web with lots of information on the prize. And Bellkor's Pragmatic Chaos web also gives inside information on the winners' road to the million.
And if you want the nasty technical details, the three teams that merged into the winning Bellkor's Pragmatic Chaos have published their solution. Here you will find a description of Pragmatic Theory's solution. Here is Big Chaos'. And here is the already well-known Bellkor approach.

After the competition ended, there have been countless reactions and discussions about it. See, for example, what Gavin Potter (a.k.a. Guy in the Garage), one of the Netflix Prize stars has to say in his blog.

So, I will add my voice to this choir of reactions to the prize. In this post I will try to summarize what, from my humble personal perspective, have been the biggest lessons learned from the prize. Take this as a warming up of the panel entitled "What did we learn from the Netflix Prize?" that we will attend in NY next week.

1. RMSE is not a valid success measure

Whether RMSE was a valid success measure for a recommender sytem was discussed very early after the prize started. As a matter of fact, this discussion is even older in academic circles.

The fact of the matter is that RMSE is not a valid success measure for several reasons. The ultimate one is that there is no direct correlation between this measure and the end-user satisfaction to recommendations. However, using more hci'ish measures related to user satisfaction is out of the question in the context of a prize such as Netflix's. It would be nice to see, though, some post-mortem in which at least the winning approach was used in a user study and compared to the original system. Hopefully, we should see a significant increase in user satisfaction with the recommendations... but, to be honest, I am not all that sure.

Once we have ruled out, user-study related measures, is RMSE the best we can do? Well, I (and many others) think that there are much better mathematical measures that correlate to the actual goal of a Recommender System. See, the main problem with RMSE is that it weighs the same the error you make by predicting a 2 where it should have been a 1 than the one you make when predicting a 4 instead of a 5. But, you would never recommend an item with a 2!

I particularly like Top-N measures such as Precision and Recall of "recommendable" items (either fixing N or, better, defining a recommendable threshold). An even more precise measure is the so-called Normalized Discounted Cumulative Gain (NDCG) where item order in the recommendation list is taken into account.

2. Time matters

Another interesting finding is the importance of modeling temporal evolution of user preferences. The fact that I liked Meatballs in 1979 does not mean that I will like Meatballs 4 now. As a matter of fact, it does not even mean that I still like the original one now. This is what we call stability in rating theory. Yehuda Koren, of the wining team, has a very interesting publication on the topic. Neal Lathia's latest publications have also interesting insights on the temporal evolution of collaborative filtering systems.

Now, it turns out that, just as you can model the importance of time, you can also take into account many other different factors. As a matter of fact, this is what the Bellkor team calls the "factor model". Again, let me point you to a publication by Koren to learn more about this (actually, now that we are at it, you might want to take a look at all of Koren's latest publications, most of which are very relevant for our discussion).

3. Matrix Factorization methods work best

Probably, the single event that marked a turning point in the Netflix competition was Simon Funk's publication of the SVD solution. Since then, many teams turned to SVD-like solutions. Matrix Factorization is the family of methods, so to say, that include particular implementations such as SVD but also many other like non-negative Matrix Factorization, Maximum Margin Matrix Factorization, and so on. Again, latest publication from Koren does a gentle introduction to Factor models in this context. The papers from the 2007 KDD cup are also a good source for information on Factor Models in the context of the Prize, since it was then when these approaches where probably thought as the ultimate solution.

Factor models are great since they can accomplish slightly better results than standard neighbor-based methods, they offer some sort of insight on the problem and, above all, they can be implemented in a much more efficient way. However, I am still to be convinced that, in isolation, they are the best method in a general case.

4. One method is not enough (nor 100)

Or, in other words, given any prediction method it usually pays off more to add a new one than to improve the existing one (If I don't remember wrong some member of the winning team said something similar in an interview).

So, yes, as sad as it may seem, there is no magical solution to the Netflix Prize. Factor models sort of work but, alone they wouldn't get you the million $. As a matter of fact, you need many, many methods combined to reach that number. The problem with this approach is that the resulting algorithms is as close to a black-box as it can get. No more insights, no more knowledge learned from it: millions of parameters that self-adjust to fit into the solution.

Don't get me wrong, this is an outstanding accomplishment from an engineering perspective. But it limits the scientific insight on the problem. It also raises the question of how portable the solution is to other domains and even datasets. I would like Netflix to pick 500K different users and 17K new movies and report the error that the system makes on them.

5. The importance of data and noise

So, to finish this list, let me bring the discussion home. As I have discussed in a talk at Boston University (and in a previous post), given a problem such as the one posed by Netflix you have two options: limit yourself to the existing data and try to bang your head to improve the algorithm, or try to improve the data itself. Of course, in the context of the prize, improving the data was not easy (although some tried to add content information to the movie titles without sucess). But in a realistic setting there are many ways this could be feasible and much, much efficient in terms of resources and results.

Take the approach to removing noise we propose in our upcoming paper, for instance. As we present in that paper there are improvements above 10% that can be accomplished by simply asking some users to re-rate some items. In another paper we also found that simply re-ordering the items to rate reduced inconsistent ratings and therefore helped in predicting recommendations.

----

As a final note, I think that the Netflix Prize has left more questions than answers while putting the spotlight on Recommender Systems research. This is of course great news for us researchers in the area. We can only hope that the 2nd edition of the prize, already announced will bring more glory to the field :-)

Let me also congratulate the winners, runner-ups, organizers and other participants. In case it was not clear: you did an awesome job!

Please post your other lessons learned as comments to this post.

Towards Context-aware Recommendations

2009-09-21T14:47:00.000-07:00

The goal of a Recommender System is to model the users' preferences in order to recommend new items that the users is likely to find of interest. However, we know that user preferences are influenced by a contextual conditions, such as the time of the day, mood, or current activity, but this type of information is not exploited by standard models. Context-aware recommender systems (CARS) aim at improving user satisfaction to recommendations by tailoring these to each particular context.

Context-aware recommendation is a research hot topic since it bridges the gap between recommender systems and other areas of research such as ubiquitous computing. However, research on this topic is still on its first stages and there is a lot to be done. If you want some background reading, I would recommend start by looking at the work of Gedas Adomavicious and Alex Tuzhilin.

I have been meaning to work on contextual recommendations for some time and this summer I had the perfect opportunity since Linas Baltrunas, a student of Francesco Ricci working on this topic for his PhD, has been collaborating with us in the lab.

In this first approach to contextual recommendations we have tried to tackle the issue of time-dependent music recommendation. That is, designing a recommendation algorithm that can recommend not only personalized music but one that fits better to the current time of the day, day of the week, or season of the year. Our initial assumption is, of course, that music taste depends on those variables (i.e. you don't listen to the same kind of music on saturday evening than on monday morning).

There a couple of things with this use case that make it specially interesting (and dificult) when compared to previous work. First, music preference modeling is done through implicit feedback. That is, users don't tell you explicitly what they like or don't, they simply listen more to some music than other. Converting that to a user preference model has some issues of its own, especially if you need to take into account contextual variables such as time. And also, time is a continuous context variable. All previous work on contextual user modeling and recommendation is done using discrete variable such as who you are with or whether you are at work or at home. This raises another issue since there is a need to segment the data before building the context-aware preference model.

We are publishing our very first initial results in the CARS workshop during the 2009 Recsys Conference, in a couple of weeks in NY. Here you have the paper. Comments and suggestions are welcomed!

Rate it Again

2009-08-05T02:29:00.001-07:00

A few weeks back, I described our work on trying to measure Natural Noise on user feedback. This was motivated by a study we recently published in the UMAP conference. One of the possible solutions that I commented then was to address the noise issue directly by trying to apply denoising algorithms to the user data.

Well, that is exactly what we have done in a paper that has been accepted for publication in Recsys09 (NY). The paper is entitled "Rate It Again" and you can access a preprint copy here. The basic idea in our approach is to ask users to re-rate items that they already rated in the past. We can then denoise ratings that prove to be inconsistent by minimizing their contribution to the recommendation process.

The biggest practical issue with the approach is that we don't want all users to have to re-rate all items in order to identify which ones to denoise. That is why in that same paper we propose ways to decide which items and users are most likely to introduce noise in order to have only those go over the burden of re-rating items.

We measured relative improvement in terms of RMSE up to 14% and we verified that this is consistent regardless of the particular recommendation algorithm (item and user-based CF, SVD, etc...).

This is another example of how to improve recommender systems using a data-driven approach. Denoising user feedback is a promising avenue, and there is still a lot of room for improvement!

Update: I forgot to mention the impressive numbers for this year's Recsys. There were 203 submisions (almost doubling last year's numbers), 140 for long papers. The acceptance rate was down to 17% making it more competitive than other 1st tier conferences. And, as a matter of fact, having been in the Program Committee, and looking at the list of accepted papers, I can say that the quality of the papers is comparable (if not higher) than the quality of recommender papers accepted to related first tier conferences.

UMAP and SIGIR 09

2009-08-04T12:50:00.000-07:00

I usually do a short report after I attend a conference. However, because of multiple commitments, deadlines and important things, I have failed to do so in the last two: UMAP 09 in Trento, Italy and SIGIR 09 in Boston, MA. So, I thought I'd give it a go and write a short report on both.

I will start by saying that lately I am not very fond of conferences in general. Don't get me wrong, I love socializing with other researchers in the area, meeting new people, and getting a chance to present my research to a larger audience while getting expert feedback. The problem is that in most conferences this comes to be secondary. There is a very interesting and recent article in ACM Communications by Lance Fortnow that does a very good job in analyzing the issue (although I should warn you that I do not subscribe his solution of going back to journals!). In any case, given this context, take my review of these two conferences with a grain of salt, I shall return to the broader issue of conferences sometime soon.

Ok, so I will start by commenting on UMAP09, which was organized in Trento (Italy) late June. There, I presented a long paper "I like it... I like it not", that I already commented in this same blog. UMAP has been organized this year for the first time by joining two pre-existing bi-anual conferences: UM (User Modeling) and AH (Adaptive Hypermedia) (see information about past conferences here). Both of these conferences where highly regarded so the resulting union was anticipated to be a success. Besides, the area of User Modeling is gaining a lot of momentum recently and a conference such as UMAP09 was expected to ride the wave. However, attendance was around 200 people, which was the same that any of the two conferences had in isolation. Acceptance rate for long papers was 26%

For people like me coming from outside the community UMAP looks like a weird conference, and there are many things about the organization that are hard to understand. First, there is the fact that the conference is not sponsored by any well-known organization like ACM, IEEE, etc... but rather by a non-profit organization called User Modeling Inc. . I am sure there are (or were) good reasons for this, but to an outsider this sounds weird, you'll give me that. Then, and possibly related to the previous, there is the issue with the proceedings: they are published with Springer (a for profit publisher) in the infamous LNCS series... and proceedings are not available for download in electronic format even at this stage! If you add the fact that the choice of location in beautiful but hard to reach Trento was questionable I am not surprised the conference turned out to be less than what the orgs expected. I really think some of these issues should be addressed soon: there are many other conferences that are more than happy to accept research related to User Modeling, and UMAP will have to do their best to attract people. However, if well-managed, UMAP should be a very attractive and relevant conference. Next year it will be organized in Hawaii and there are already talks of co-locating it with a larger event in 2011.

Moving on to SIGIR, which is a completely different beast since it is a well-established first-tier conference and it was organized in the easy-to-reach Boston. I have no complaint about the organization (except for maybe the lack of lunches during main conference days). And as a matter of fact I have to congratulate them for excellent social events: both the banquet at JFK museum and the Harbor tour were great. I presented our paper "The Wisdom of the Few" in a conference that had the lowest acceptance rate since 1997, a 16% ... quite an honour being accepted.

SIGIR -- the most important conference on Information Retrieval, for those that are not in the field -- is one of those large conferences where you have several parallel tracks. This is great since you are always likely to find something you are interested on. However, it has the downside that most people might be attracted to the same track, leaving others almost empty. Curiously enough, this is what happened during the Industry track: most people were attracted to it, leaving the research tracks with much less attendance than expected. A suggestion for next years would be not to host the industry track in parallel but during a specific time. In any case, this brings me to my important question: why where researchers more attracted to the industry track than the research ones? The answer is simple: while the average presenter in the industry track is a well-known professional with an above-average public speaking skills, the average presenter is a PhD student that can barely hope to grasp the audience attention by not putting too many formulas on the slides. I will leave this analysis here but will try to come back to it in a dedicated post soon. If you are interested in reading more there is a great series of posts on the excellent Industry Track at SIGIR 2009 by the organizer Daniel Tunkelang, Endeca's Chief Scientist. Start here, and follow to similar or later posts. You can also find other great posts summarizing SIGIR, see Jeff Dalton's summaries here, for instance.

On the last SIGIR day I attended the Search in Social Media Workshop. This turned out to be one of the highlights of the conference. The setup of the workshop was great. It was divided in different topic blocks. For each block there was a keynote. Then other presenters had a short time for presentation and then they all joined for a discussion panel including participation from the audience and from a twitter feed projected on the side... Brilliant! I particularly liked the keynotes by Joseph Konstan and Abdur Chowdhury, Twitter's Chief Scientist.

Overall going to conferences is a great experience and I got to meet many interesting people, have interesting conversations and I presented 2 papers getting a lot of feedback. Conferences are essential in the work of a researcher. However, there is a lot of room for improvement in order to make the best of them. And definitely CS conferences should take the lead because technology will be key in this transformation.

It's all about the Data...

2009-07-21T05:53:00.000-07:00

(or Data-driven approaches to the Recommendation problem)

This is the title of the talk I will be giving on Thursday July, 23, 3:30 pm at Boston University (Math & CS Building, room 135).

Here is the abstract of the talk, hope to see you around:

The Netflix Prize put Recommender Systems (RS) research in the spotlight. Given 100M ratings from 500K users to 17K movies, researchers from all over the world have been racing for almost 3 years to improve accuracy by 10% in order to win the 1M$ prize. A couple of weeks ago, it was announced that a merge between several teams that used hundreds of predictors might have won the prize. However, there are doubts about the generalization properties of this winning approach.

Our approach to the Recommender problem has been different: instead of taking data as is and invest in fine tuning a large number of machine learning algorithms to model the data, we have focused on understanding the data and improving it.

In a recent UMAP 2009 paper named "I Like it, I Like it Not..", we show that the natural noise due to the inconsistencies in user feedback sets a lower bound on the so-called "magic barrier" in RS and could in fact be very close to the Netflix Prize threshold. Once the inconsistencies of users when providing explicit feedback have been characterized, we can devise ways to minimize them.

In "The Wisdom of the Few" (SIGIR 2009), we propose a different approach to Collaborative Filtering by using feedback from experts instead of regular users. In "Adaptive Data Sources" (ITPW @IJCAI 2009) we propose to use ensembles of data sources instead of ensembles of predictors. Finally, in "Rate it Again" (RECSYS 2009), we present an algorithm for denoising user feedback based on a re-rating approach.

In this talk, I will give an overview of the issue of noise in user feedback for Recommender Systems and will briefly describe the work that we have done (as previously described) to overcome it.

Why the crowds are not (always) wise

2009-07-13T01:37:00.000-07:00

More often than not I come across situations in which the now famous "Wisdom of the Crowds" is applied in the wrong context or situation. Let me try to explain it with an example of one such situations:

Let's imagine a well-known online newspaper decides to give a prize to the best... Linux application, for instance. In order to do so, it decides that it will let the crowds decide. First, developers who have an application and are interested in the $100K of the prize will submit their application. Then users will vote and the prize will be awarded to the application with most votes.

Ok, so, do you think the winner of the contest will be the "best Linux application"? Of course not. And there are many possible reasons why that won't be the case, right?

Let's start by the two most important conditions that we need to guarantee if we want to trust the Wisdom of the Crowds:

There has to be... well, a crowd
Whatever we are measuring needs to be close to the notion of "popularity"

Ok, the first condition seems fair and obvious enough. However, it is often forgotten. If you want the crowds to decide on something, you need to have enough opinions to avoid possible bias. Ideally, you would even worry about things such as demographics, etc... Given that this is seldom possible, you need to guarantee that malicious bias or shilling is not possible, or at least hard.

For instance, in our example, imagine the winner Linux application got 120 votes. "Uhm", I hear you say, "for $100K, I could get 120 people to vote my app". Well, there you are. And there are many cases in which this might not be so explicit, but in which the amount of opinions do not make a crowd.

Ok, let's go to the second condition: we need that whatever we are measuring is somewhat correlated with popularity. In our example, it might be that what we mean by "best application" is precisely the "most popular" one. If so, that's ok. If what we are trying to measure is something else: such as "most innovative", "most secure, estable, ...." chances are that the crowds will not be evaluating these features. Again, in many cases, what the "best" means, is not critical (think on TV programs such as American Idol where the crowds get to pick the best performer, for instance). However, in some others, we might be getting an answer to the wrong question.

Although I summarized/simplified the two most important conditions for the Wisdom of the Crowds to take place, the original book by Surowiecki talked about 4 conditions:

Diversity of opinion
Independence
Decentralization
Aggregation

(The Wikipedia entry does a good at summarizing these.)

Wisdom in crowds should be taken with care and only where appropriate.
Actually, in many situations, instead of crowds we might prefer to have a number of experts give us their opinion (would you post your symptoms on a web page and decide what pills to take based on the wisdom of the crowds?).
For this reason, we propose a Collaborative Filtering approach based on experts in our forthcoming SIGIR paper (see my previous blog entry on this).

And for that same reason, I am starting to like the concept of the Alchemy of the Crowds (as opposed to Wisdom) described by the authors of the same name.

Sometimes your neighbors might not be your friends

2009-07-03T04:39:00.001-07:00

A few days ago in UMAP 09, Rachael Rafter and others from Barry Smyth's group from the Clarity Center presented an interesting paper entitled "What Have The Neighbours Ever Done for Us? A Collaborative Filtering Perspective" (See abstract at the bottom of this page**).

The paper draws its main conclusion from several studies over several datasets. And what it has to say may sound surprising: in standard nearest-neighbors collaborative filtering, more often than not, neighbors are contributing negatively to the prediction. In other words, using the user mean would work better in many cases than adding the contribution from the neighbors. The problem is that this gets even worse when looking at precisely the most informative ratings (i.e. those at either extreme of the scale).

These findings seem to be somewhat related to the paper we are presenting next week in the The 7th Workshop on Intelligent Techniques for Web Personalization & Recommender Systems at IJCAI (well to be precise I should say Neal Lathia will be presenting it).

In that paper, we are presenting a novel approach to finding nearest neighbors: instead of looking at users that you have close by we try to find your most like-minded peers looking at several different datasets recovered from the web. Maybe your movie watching taste (and rating style) is better predicted by a group of experts. Or maybe it is a group of power users from a completely different dataset the ones that fit you better. In our paper we use only three datasets: Netflix, Flixter, and Rotten Tomatoes.

We found that if we had a perfect classifier that could tell us what group a user belongs to (or will be better predicted by) we could improve the accuracy level way below the Netflix prize threshold. The novelty of this approach is that now, instead of focusing on merging many complex algorithms, "all" we need to do is to treat the recommender problem as a classification problem and use algorithms as simple as we want.

So, I am hearing you ask: where is the catch? Well the catch is that we haven't been able to find a good classificaiton algorithm (in that case we would be talking about a long paper in a conference not one in a workshop). We have tried a bit with several simple classifiers (including SVM, NN, etc...) but there is no way. However, there are many promising avenues including cost-sensitive classifiers.

So, if you are an expert on classifiers and want a fun project, join us in finding who your good friends are... beyond your neighbors.

** Unfortunately I can not point you to any digital version of the article since there is none available. Just another of the bad choices by UMAP organizers is to publish proceedings with Springer and not clear rights for a digital version to be available right away.

The (very draft) Agile Research Manifesto

2009-06-15T02:40:00.001-07:00

Some time ago I posted about the idea of applying agile principles to Scientific Research. The idea is built upon the observation that the Scientific Process is iterative by nature and shares many ideas with Agile Methods. I have used agile-like methods to coach students and small research teams with very good results.

After the previous post I have been discussing these ideas with a few people. One of the things that came to mind is that it should be possible to adapt the Agile Manifesto without making substantial changes to its underlying principles. I have recently published my first attempt at doing so. You should treat this as an early draft up for discussion and feedback (As a matter of fact, the version you will now read already includes feedback from some discussions). And, if you would like to participate in the discussions, please join the LinkedIn group we created for that purpose.

Netflix Prize: What if there is no Million $ ?

2009-05-24T16:29:00.000-07:00

If you are participating in the Netflix Prize, don't worry... This post is not about the economic crisis and Netflix filing for bankrupcy and not paying the prize. But this is in fact about a much more "scary" perspective: what if there was no way to lower the threshold set in the competition? Or more precisely, what if the only way to lower the error threshold was actually overfitting to the existing training and testing dataset?

The possibility cannot be discarded. A few weeks back I posted on this same blog a discussion on a recent work we did analyzing the impact of natural noise on user ratings. That is, when users give us feedback through ratings, they are adding a background noise. There are many possible reasons for this. In some cases the user does not really see a difference between rating an item with a 3 or a 4. Other times, the user is not being careful enough when giving the feedback and is letting other factors affect the result. Ratings will be affected by things like how long ago the item was used, what was the previously rated item, or even the mood the user is in.

But, whatever the reason is, the result is that we have data with noise and/or errors. If we take a random rating and asked the user "What was your rating for item X?" we will inevitably get errors. Uhm... so even the user makes errors when recalling her own ratings? Yes! As a matter of fact we can easily measure this error by asking her to rate the same items several times (see, again, my previous post on this).

Now, here is a rule of thumb: we cannot predict the user any better than this same user can assess her own ratings. Therefore this natural noise threshold is setting a "magic barrier". We cannot try, and it really makes no sense, to go below this error in our predictions. What difference does it make that our system is very good in predicting some item should "be" a 3 if the user does not really see the difference between (or is not sure about) a 2 and a 3 for that item. Or, the other way around: "How can we predict a 3 with no error if the user "randomly" moved between 2,3, and 4 when giving us feedback?"

So, and returning to the initial issue, the question is now: is the Netflix Prize threshold below this "magic barrier"? Do they even know? Well, a member of the leading team Korbell and I had an informal conversation with Netflix VP for Personalization. Of course, they cannot say much about the prize in case they would be giving vital information for winning the price. However, when we asked him whether he had any information as for what was this "magic barrier" on the Netflix Prize dataset he answered that they did a small study to estimate something similar to this. Their estimation was "around 0.5". That is surely non-negligible but it is safely located below the winning threshold of 0.83. However, remember this was only a small study that gave them a rough estimate. Our measures on a similar dataset yielded RMSE values between 0.57 and 0.82 Although these values depend on several variables such as the time between ratings or even how items are presented to the user, we have reasons to believe the Netflix dataset should be on the higher end of this range (if not higher!). Read more on our UMAP article.

As a final appendix, let me throw in two important conclusions that pinpoint future directions. First, it is clear that the RMSE measure should be reconsidered. If, on average, the user does not know the difference between a 2 and a 3, we should not take that into account in our success measure. Top-N measures seem much more suitable as a measure of success in Recommender Systems: the user might not care or see a difference between a 2 and 3, but she will surely be deceived if we recommend something she values with a 1.
Another strategy is to select only users that are more consistent and use those to generate recommendations for the target user. If the target user is noisy herself, we will still get lousy recommendations. But we will be minimizing errors for the rest. This is the approach we took in our Wisdom of the Few. Finally, although we cannot aim at getting results below the "magic barrier", there is something we can do: lower that barrier. In a work we have under submission, we devised a "denoising" algorithm that is able to improve accuracy almost up to a 15% by lowering this noise threshold. But, I will leave this for a future post once we hopefully get the paper accepted.

Allosphere@TED (part II)

2009-05-24T16:09:00.000-07:00

In a previous post I talked about the presentation my former boss (and still friend) JoAnn Kuchera-Morin gave at TED talking about the Allosphere project I was technical director at UCSB. I am now writing this follow-up post because since then many more people have found about the project.

TED put the video of the presentation online a few weeks ago.

Shortly after, the video was slashdotted. And of course, once that happen you are likely to get much attention. Some of the most interesting comments I found:

AlloSphere three story virtual environment not available for birthday parties @ Endgadget
Singularity Hub
The AlloSphere: Flying through a giant virtual brain?
HCI and the Allosphere
AlloSphere: Interpret Scientific Data in a 3 Story High Metal Sphere

I particularly enjoyed this one and this other one. Funny, when I was working on the project people made the joke about Professor X's Cerebro, where I was the X :-)

Hopefully, now that the project has caught more attention, it will be easier to get the right money in the door.

ACM Fellows 2008

2009-05-24T14:51:00.000-07:00

A few weeks ago I read the list of the 2008 ACM Fellows. ACM each year recognizes computer scientists for their contributions. The 2008 list includes 44 new fellows from very different backgrounds. However, I was happy to find out that I knew some of them and I definitely agreed with them being on the list. I will add my small grain by mentioning them in this blog:

Alan C. Kay
For fundamental contributions to personal computing and object-oriented programming

The surprise here is how in the world could Alan Kay still not be an ACM Fellow! I can thing of maybe only a handful of people that I would put before him in such a category. If you are doing anything related to computers you probably already know a lot about Alan: father of Object-Oriented Programming (including Smalltalk), inventor of laptop, the windows-based interfaces, the OLPC project...
Still, this is a good excuse to read and learn a bit more about him. Alan was one of my usual cites in my Software Engineering lessons and I was fortunate enough to meet him during the presentation fo the OLPC project in UCLA and learned about many things, including his love for Spanish ham and his current relation with several Open Source projects.

Perry R. Cook Princeton University

For contributions to computer music, physics-based sound synthesis and voice analysis/synthesis

I have known Perry for many years. He was even a professor in one of my PhD courses at UPF. But anyone who has done some research in anything related to computer music knows Perry Cook. He is most known for his work on physical modeling of instruments and voice synthesis. But he has also co-authored very important sofware packages such as STK, which was a big influence on our CLAM. More recently, he has also coauthored Chuck with Ge Wang. But above all, Perry is a great guy... someone you will want to hang out with after the conference and have some beers.

Joseph A. Konstan - University of Minnesota

For contributions to human-computer interaction

Funny that I am hosting Prof. Konstan this Friday on his visit to Barcelona. I have just met him briefly in previous ACM Recsys conferences. And to be honest, I was not very much aware of his previous work. Joseph was well-known to me for being one of the founders (together with John Riedl) of the GroupLens research group. Their work on Recommender Systems has been seminal and extremely important for raising awareness of this field in recent years. But apart from that it turns out that Prof. Konstan has been President of SIGCHI (one of ACM's most important Special Interest Groups with 4500 members). He is also known for his work on Online Communities and Computer Systems for HIV prevention. Read more in his webpage.

William Buxton - Microsoft Research

For contributions to the field of human-computer interaction

Bill Buxton is an amazing guy I had the pleasure to meet in Santa Barbara. That was before he was appointed as Microsoft Chief Scientist. But still, he was teh kind of person everybody listened to as soon as he started talking. You only need to read his bio tu understand why that is. He started working also on Computer Music and at that time started working on multi-touch surfaces and composition tools. He then went off to be Chief Scientis at Alias/Wavefront, now part of Autodesk and known worldwide for their Maya package. It is a bit weird to see him now as Microsoft Chief Scientist. But, hey... that's a heck of a job!

Linux: Processing audio from the browser

2009-05-21T15:12:00.000-07:00

Imagine being able to process streams of audio that are playing on your browser directly and on real-time on a external application. For instance, you could analyze the music of a youtube video while it's playing.

Well, this is possible in Linux with a little infrastructure: jack, pulseaudio and a couple of modules that connect one with the other. This post in the ubuntuforums gives a good enough explanation on the requirements (scroll down until you see the section on "pulseaudio through jack").

Anyway, I have recorded a two-part screencast where I explain all this and use this setting to process a youtube video with CLAM and detect its chords in real-time.

Part 1

Part 2

(Final demo step, recorded with a camera given to the problems using jack, clam, and the gtk-RecordMyDesktop app at the same time. Still... you'll get the idea after watching part 1)

The Wisdom of the Few

2009-05-17T15:21:00.000-07:00

One of the most common approaches to Recommender Systems is the so-called Collaborative Filtering. The main rationale is the following: In order to predict items that you will like, we find the most similar users to you by looking at your previous likes and dislikes. We then recommend items that those users have liked, but you still don't know.

There are several caveats with this approach. One of them is that we need an effective way of capturing users likes and dislikes. Most of the times we need to do this by asking users to explicitly rate items. This is the typical 1 to 5 star rating that you get in many services from Netflix to Amazon. But we know, as I commented in an earlier post, that users are noisy when giving that feedback.
So, because rating feedback is noisy, we are prone to make errors when predicting what a user likes or doesn't like.

But, standard Collaborative Filtering has several other problems. First, because we need to compute neighbors and predictions, we need to transmit all user ratings to a centralized server and this can compromise user privacy. The number of users and items is likely to be huge and applying this approach is computationally expensive and has scalability issues. And so on...

We have proposed a new approach called "Expert-based Collaborative Filtering". In this approach, instead finding neighbors from a general pool of like-minded users similar to the target, we find neighbors in an expert database. The rationale is that these experts will be much more consistent in their ratings (i.e. less noisy) and data will be less sparse.

We have conducted experiments using movies and experts from Rotten Tomatoes and concluded that users prefer recommendations drawn from like-minded experts more than those predicted from (noisy) like-minded peers.

In the next SIGIR 2009 conference in Boston we will be presenting the paper entitled "The Wisdom of the Few: A Collaborative Filtering Approach Based on Expert Opinions from the Web". Here you can access a copy of the paper where you will find a complete explanation about this new approach.

Update: here are the slides I presented at SIGIR

The Wisdom of the Few @SIGIR09

View more presentations from Xavier Amatriain.

WWW 2009 Conference

2009-05-17T14:27:00.000-07:00

I have been meaning to blog about the WWW conference since it happened a few weeks back but have been pushing it back because of deadlines. In any case, I did not want to let it go by without at least writing a few lines.
The WWW09 conference was held in Madrid, April 20-24. I have attended many conferences in my life but this was the first time I was in a WWW conference. And it was overall a really positive surprise. The WWW is a very large conference. Most of the time there are up to 8 parallel tracks in the main conference, let aside posters and other events. However, the organization was extremely good (a german colleague joked that it did not seem a spanish-organized event).

Probably one of the highlights in terms of the organization was the visit of the Prince of Spain for the opening ceremony. Although he speaks perfect English, he gave a talk in Spanish because of protocol rules... weird. You can see my (very bad) recording of the opening ceremony here (part 1, part2, part3).

The conference was really taken by the twitter hype. It was amazing to see people tweatting at all times and about everything. You only need to do a search for #www2009 in twitter to find out (it was a trending topic for a large part of the conference). Or you can see this amazing picture of people tweating and blogging during a Flamenco concert in the conference reception.

One of the interesting surprises of the conferences was the Developers Track. One of the bad things about conferences talk that I have complained in the past is that they seldom add anything beyond reading the conference paper. But in the Devel Track, this was not true in anyway. There were very good presentations including demos, in-depth explanations with code examples, etc... This is the track where I presented the work result of Jun's GSoC that I already explained in a previous post. Most of the talk was recorded and is now available on three Youtube videos (part1, part2, part3).

Other interesting presentations from the Telefonica Research team include Pablo Rodriquez' keynote, and Josep M. Pujol's presentation of the Porqpine search engine.

I am usually not very fond of very large conferences such as the WWW. But I have to say that I really liked it. Having so much to choose from guaranteed that there was always something interesting to attend. And there was a great balance between academics, hackers and people from industry. Definitely, one of the conferences I want to be targetting in years to come.

Multilevel Audio Descriptor Aggregator @WWW09

2009-04-13T14:38:00.000-07:00

If you take a look at the Developer's track in next week's WWW Conference, you will see an article in which I am listed as a co-author: "Combining multi-level audio descriptors via web identification and aggregation".

This is mostly the result of Jun Wang's work with CLAM in last year's Google Summer of Code. Of course, a bit more of work followed after those 3 months but the bulk of the work was done on the Google grant. It must be said also that Jun is an exceptionally brilliant and hard-working student (Chinese Academy of Sciences).

The paper explains how CLAM has been used to build a tool that is able to integrate many levels of audio descriptors - ranging from low-level descriptors extracted from the audio signal itself to high level descriptions gathered from the web.

In Jun's blog, you can read a longer explanation including a video clip of the CLAM Aggregator at work.

Recsys 2010 in Barcelona

2009-04-06T16:08:00.000-07:00

It has just been announced: Recsys 2010 will be held in Barcelona! And Marc Torrens (Strands) and myself will be sharing the general chair of the conference.

ACM Recsys is the premiere annual event for researchers and practitioners on Recommender Systems. This year, the 3rd edition of the conference is taking place in New York... a great preparation for next year's conference in Barcelona.

I have organized conferences before both in Barcelona and Santa Barbara and I know how much work is needed. But this is an excellent opportunity and we are sure we will be hosting an excellent event.

"I like it... I like it not" or How miss-behaved users are when giving feedback

2009-04-02T15:34:00.000-07:00

Many recommender algorithms base their accuracy on the assumption that users are able to provide a good assessment on their preferences. This is known as "explicit feedback": users explicitly rate items and those ratings are used as the base to create their profile. The Netflix prize, for instance, assumes that we can predict user ratings to an acceptable level of accuracy by simply taking past ratings into account.

However, it is a known fact that users are fairly inconsistent when asked whether they like or not a given item, let aside the idea of giving an accurate rating on a 1 to 5 scale. Much work has been invested in recent years to come up with better prediction algorithms but, in general, these algorithms disregard the existence of such noise in the user feedback.

In a recent work, accepted to the UMAP 2009 conference, we focus on analyzing this noise in user feedback. In order to do this, we devised a two part test-retest experiment in which we had users rate the same movies three times. We measure an RMSE which is roughly between 0.5 and 0.8. This value is related to the "magic barrier" in recommender systems (i.e. the minimum error a recommender system can accomplish in practice).

Other findings include the fact that extreme ratings are more consistent than those in the middle of the scale (see figure) ; and the order in which movies are presented to users affects the consistency of the rating. Read the full paper for more details... and come to Trento for the UMAP conference for the presentation.

The Barcelona Startup Scene

2009-02-15T13:42:00.000-08:00

For some time now, Barcelona is trying to present itself as the "European Sylicon Valley". But, to contradict Techcrunch, there are many more nice things to Barcelona than its climate. First, there is an large population of very talented techies. Surely, this is in part due to the good level of the local universities (what can I say... I teach in one of them :-). Second, local people are known for being creative and open-minded, and you can breath some of that in the Barcelona air. And finally, because of its "charm", Barcelona attracts many talented and creative people from abroad. Given a choice, they'd rather work on their laptop from the sunny Barcelona beach than from a rainy place in northern Europe, I guess (so, yes... it's about the weather).

Local government agencies try to support start-up through different programs. However, life is not easy for new tech companies in Barcelona. The main issue is the almost non-existent local venture capital. As Mario Nemirovsky put it in a recent conversation I had with him: there is capital, but not venture. On top of that, being a Spanish-based start-up does not make things easy to get to foreign VC money (especially US). Not to mention the excessive bureocratic burden put on new companies here in Spain.

However, there are many people that are definitely putting their share to change this and make of Barcelona the real European Sylicon Valley. I will review some of them in the next paragraphs. I admit the following review is biased towards people I personally know, many of whom are actually friends. So if you feel you should be here, let your voice be heard!

Mario Nemirovsky, whom I mentioned before, is one of these. Mario has founded several companies, such as ConSentry, in the US. He is considered one of the most influential latinos in Sylicon Valley. He is currently in Barcelona on a grant but his main interest is in briging the Barcelona startup scene to the next level. As a first initiative, he helped found Miraveo. Miraveo offers a really interesting and revolutionary approach to create spontaneous adhoc wireless networks.

Also based in Barcelona, Terry Jones (@terrycojones) and Esteve Fernandez are about to release the first beta of FluidDB in their Fluidinfo mind-blowing startup. Fluidinfo has been named as "the next Google" or "world-changind" by people such as Robert Scoble or Tim O'Reily. It is hard to describe Fluidinfo in a few words so I will use their "Database meets the Wiki" lema. But if you really want to understand what they do, check the four videos in this Scobleizer post.

3Scale is another Barcelona startup that I know pretty well. Actually, it is the only one of the ones I mention in this post where I have put some (very little) money. 3Scale provides a solution to manage all issues related to a web service. Their product is already fully functional and they have a great business model. They were selected for Techcrunch 50 and Le Web Paris this year. Need I say more? :-)

BMAT (Barcelona Music and Audio Technologies) is a spin-off from the MTG Research group where I did my PhD. And therefore, I have many friends in there. They have a number of B2B products related to audio and music such as music search and recommendation, voice processing and games, or broadcasting monitoring.

Strands also started being a music-oriented company but is now much more than that. Apart from their public product, they build custom solutions for recommendation and social networks. Curiously enought, the startup was originally started in Corvalis (OR) by Francisco and Marc, from Barcelona. But now that the company has grown, they have moved much of their activity to Barcelona (although they are still active in Corvalis and several other locations around the world).

I should also highlight Fluendo and Flumotion , companies that started out of the great GStreamer open source project (which, by the way, is closely related to our own CLAM project).

I will finish by mentioning a couple of the startups from serial-enterpreneur (and friend) Otto Wust. After founding Sclipo and getting it to win the European 2.0 startup in 2007 and making it to the RedHerring top 100 list in 2008, Otto is now focusing in a new venture. NicepeopleAtWork is offering a complete solution for video production suite for publishing web videos. Although the company is very young, they already have interesting results and important clients.

Ok, and I know I am missing many more so I might be doing a Part II of this post. In any case, let your voice be heard through the comments.

Something is moving in Scientific Peer Reviewing

2009-02-15T09:54:00.001-08:00

During these past few months I have had pretty bad experiences with peer reviews at conferences: unfair reviews, others that were plain wrong,
papers considered off-topic because I did not respect what would be expected by the "community", poor or non-existent intervention of the track or PC-chair... I guess this will all sound familiar if you are in the research business. Being on the other side (the reviewer's) does not help to see things much better.

There are many people now thinking that the system we have nowadays is flawed in many ways and probably things are going to change (hopefully for the better) soon.

Along these lines, Michael Nielsen had an interesting post in his blog. He talks about the three myths of scientific peer review: (1) Scientists have always used peer review; (2) peer review is reliable; and (3) Peer review is the way we determine what’s right and wrong in science.

Jon Crowcroft, S. Keshav, and Nick McKeown also write an interesting article in the past issue of Communications of the ACM. The article, entitled "Scaling the academic publication process to internet scale", proposes a way to use Web 2.0 paradigms, and in particular crowdsourcing, as a way to overcome the flaws of the current system. The goals of their proposed process are stated as: (A1) Authors should not submit poor papers; (A2) Authors should become reviewers; (R1) Reviewers should submit well-substained reviews; (R2) Reviewers should not favor their friends; and (3) Reviewers should not denigrate competing papers. Although many things are still left out of their analysis, it does seem like an interesting and promising step forward.

It is really interesting that just as I was reading this article I found out about Google's gPeerreview. According to Google, they "intend (...) to do for scientific publishing what the world wide web has done for media publishing".

Again, it is clear many of us think things are not working the way they are. Therefore I can only applaud any initiative that brings us closer to a more fair and sustainable system.

WSDM 2009

2009-02-15T08:38:00.001-08:00

Last week I attended the Web Search and Data Mining (WSDM '09) conference in Barcelona. The conference was organized by Ricardo Baeza-Yates and the people at Yahoo Research Barcelona, and it is always nice to attend a top research conference without having to take the plane.

If you ever wondered what researchers at the big internet players do, this is your conference. As a matter of fact, 66% of the accepted papers had an author from Google, Microsoft or Yahoo. Given that the acceptance rate was pretty low (16%) this might come as a surprise. However, not if you think about how much research in the area of Web Search this companies have (probably more than 66% of the researchers in the area work in these three companies, especially if you count research internships).

If you look at the program, you can see that the hottest topics seem to be, arguably: personalization, tagging, and link and click analysis. The best paper went to Fernando Diaz (Yahoo Labs Montreal) for an interesting work in integrating news content into search results. Here, you can read his paper (btw, all papers will be available for free download, it seems).

The highlight of the conference, however, were for me two of the keynotes. First , Google's Jeff Dean, of Map Reduce and Big Table fame, gave an amazing talk about the evolution of Google's systems and architecture in response to ever growing demand. After his talk I was wondering what percentage of Google's success could be directly linked to their systems development strategy (versus algorithms, interface, and business model).

On the last day there was a keynote shared with the colocated WAW conference. Ravi Kumar, from Yahoo, gave an interesting overview of his research in Social Networks. I enjoyed his talk. However I was a bit annoyed by how lightly the idea of "causality" is used in some of these analysis works. "If you have an obese friend you have 25% more chances of being obese", said Kumar refering to an earlier work by sociologists. This confussion between co-occurrence (or homophily) and causality is a bit scary, especially coming from someone like him that has actually worked on the issue. His attempt to model causality was later described. Using the so-called (random shuffling test) he interpreted temporal precedence as causality. Yet another arguable interpretation.

Finally, I have to say that I felt most talks were rather on the "boring" side. This is probably nothing to blame WSDM for. It is rather my feeling that talk-based conferences are challenging my short and volatile attention span, especially if I have a laptop on me. I will post more on this soon.

Allosphere@TED

2009-02-15T08:12:00.000-08:00

The Allosphere, the project I was coordinating a UCSB, was presented last week at the TED talks. In case you are not aware, the TED talks is something worth knowing about. TED stands for Technology, Enterntainment and Design but, since it started in 1984 its scope has become even broader. In short, TED talks aim to bring together visionary people to discuss the future. Talks verse about many different things, and every year there are a few that make headlines. This year, for instance, Bill Gates threw some mosquitoes at the audience to make his point on Malaria in the third world.

JoAnn Kuchera-Morin, director of the Allosphere, was able to give a short 3 minute talk about the project. And, although 3 minutes may seem a really short time, doing this in such a setting has really brought a lot of buzz around the Allosphere in the media.

Some interesting posts related to the Allosphere at TED:

At the Wired blog
The Allosphere at CNET
GreyNYC talking about the Allosphere
Post about the Allosphere at the Art of Science
The Allosphere mentioned at the Harvard Business Blog
A post in "In my copious free time"
An interview with JoAnn before the TED talk at NowPublic
Even a post in Spanish by Diego Leal

Hopefully all this buzz will help the project get the funding necessary to finalize such a visionary and amazing idea.

Open Research Day

2008-11-16T15:07:00.000-08:00

A couple of weeks ago we had our Telefonica Open Research Day. Finally I found some time to blog about it.

It was a half-day event in which we had both talks and demos showcasing the latest developments in Telefonica's Scientific teams.

In the talks we had several invited speakers: Mateo Valero, Head of the Computer Architecture Department at UPC and Director of the Barcelona Supercomputing Center gave a talk on the "Future of Supercomputers"; Sandeep K. Singhal, Product Manager of the Windows Network Team gave a talk on the "Challenges of Networking in the 21st Century"; Federico Casalegno, head of the Design Lab at the Massachusetts Institute of Technology talked about their projects related to social mobile and information sharing; and Arturo Azcorra, Universidad Carlos III and IMDEA Networks talked about Internet 2.

Then we had our Multimedia Scientific Director, Nuria Oliver, talk about the challenges related to the explosion of content, seamless connectivity, and decreasing attention span from users. And Pablo Rodriguez, our Internet Scientific Director, talked about using the network as a FedEx service to ship bulk data from one point in the globe ot another.

I took part in a panel entitled "Search, Recommendations, and Personalization: Text and Beyond" where we also had Hugo Zaragoza from Yahoo Research, Xavier Serra from the Music Technology Group at UPF, Ferran Marques from UPC, Marc Torrens from Strands, and Alejandro Jaimes, head of the Telefonica Scientific group on Datamining and User Profiling. Xavier Serra and Ferran talked about ways to bridge the semantic gap in multimedia - in the case of music and images, respectively. Hugo talked about the challenges in Search and Marc on the power of Recommendation. Alejandro talked about how culture should be taken into account in applications and algorithms that deal with people. I gave a talk on how Recommender Systems can become an alternative to Search engines (you can find my slides here).

You can find all the slides for the talks here.

Then we had a bunch of interesting demos from many of our research projects. Josep M. Pujol and his Social Search Engine prototype; Joachim Newman and his project analyzing Bicing users behavior; Xavier Anguera presenting our project on multimodal interfaces for picture browsing on the cell; the This or That project on social interaction over the cellphone and Facebook for shopping in conjunction with MIT; Xiaoyuan Yang's Kangaroo P2P solution for video broadcasting; several projects on wireless, network...

Overall a quite successful event with over 120 people from different backgrounds (universities, industry...) completing a full house.

Let me know if you need further information on any project or you would like to be included in next year's list of guests.

ACM Multimedia 08

2008-11-05T15:08:00.000-08:00

My next stop was in Vancouver for the ACM Multimedia conference.

Vancouver is an awesome city and I was fortunate to have two friends (Alberto and Juan) living there. So I was able to attend the conference, have fun, and eat wonderful sushi, all at the same time.
The first two days I was invited to the ACM SIGMM retreat. Around 35 top researchers from around the world gathered to discuss on the future of Multimedia. We discussed issues related to research, education, and industry relations. Overall this was probably the best part of the conference as I got to meet and talk at length to very interesting people. Socializing in such a setting was much easier than in the typical, and overcrowded, coffee breaks. (In the picture Nicolas Georganas and Wolfgang Effelsberg, chairs of one of the breakout sessions).

Outside of the confernce, I also got to meet Tim Bray, currently director of Web Technologies at SUN and well-known for being one of the fathers of technologies such as XML and RDF. I admire him for this and for his views on Agile development and Open Source. I was delighted to hear from him that he is completely in favor of REST and does not believe in the Semantic Web.

The conference itself was pretty good although I have to admit that I got much more out of demos and posters than out of regular paper presentations. To be honest, I have come to think that most presentations in conferences are a loss of time. Unless you have a good presenter (and that happens around 10% of the time) you are better off reading the article. Demos and posters, however, are different as they offer a one-to-one interaction with authors.

The Open Source prize was sponsored by us (Telefonica Research) and the winner was the Network-Integrated Multimedia Middelware, a pretty amazing piece of sofware for addressing multimedia devices over the network. It is great to see such awesome projects getting the price we won in 2006 for CLAM.

Recsys 08

2008-11-05T14:40:00.000-08:00

Last week I attended the 2nd ACM Conference on Recommender Systems in Lausanne. Regardless of being just in its second edition the Recsys conference is already showing signs of becoming a top tier conference very soon. 121 submissions (a 100% increase over the first edition) and a 31% acceptance rate (that is including short papers and posters) are indeed very promising figures. Of course this brings together an overall increase in quality in all accepted papers.

Another highlight of the conference were the three very interesting tutorials on the first day. Robin Burke talked about robustness in Recommender Systems. Yehuda Koren, of Netflix Prize world fame, gave an interesting tutorial on the approach they are using for staying at the top of the Netflix Challenge leader board. Yehuda has now left AT&T and joined Yahoo Research in Israel so it is unclear how much he will continue working on the prize from now on. Finally, Gediminas Adomavicius talked about context-aware Recommender Systems in another very interesting tutorial.

One of the interesting differences in Recsys in relation to other conferences in related fields is the high percentage of industry participants (around 50%). The guys from Strands have been doing an amazing job of making sure this does not become a purely academic conference. This year they even offered a $100,000 prize for the best start-up idea related to Recommender Systems. The winners, also co-leaders of the Netflix prize, presented an IPTV recommender, which reminded me a lot of some of the work we are doing in Telefonica R&D. You can read more about the prize in Strands' blog.

At the end of the conference we made a bid to bring Recsys to Barcelona in 2010 (next year is in New York). If we get selected I will be co-chairing the conference with Francisco Martin, Strands' CEO. Stay tuned for more info on how this develops.

Back to teaching

2008-10-08T05:18:00.000-07:00

A couple of weeks ago the course started again. After some minor seminars last year I have gone to "real" teaching at the UPF university this year. I will teach some classes in the Information Retrieval class and I am responsible for the Software Engineering course to 3rd year CS students.

I used to teach this course before going to the US and going back to it is really enjoyable. The syllabus is divided in two parts: in the first one we cover issues related to the Software Process, Methodologies, and Requirements Engineering; in the second one we focus on advanced object-oriented analysis and design.

On the practical side, students do a complete life cycle simulation starting with requirements engineering but quickly goint into iterative development including a big focus on test-driven development and patterns.

This year I have pushed some more content on the "methodologies" side. We focus quite a bit on Agile methodologies (especially Scrum and XP) but also study things like RUP or CMMI.

Here is the course website with lots of materials (albeit all of them in Catalan)

Postdoc position in Recommender Systems

2008-10-07T16:13:00.000-07:00

Looking for people to work with, maybe you are interested...

The research group on Multimedia in Telefonica Research* Barcelona invites for applications for a Postdoc position in the area of Recommender Systems.

We are looking for dynamic, creative, and resourceful individuals to join our research efforts in designing the next-generation of Recommender Systems and Algorithms. Our research impacts all areas of the company, including projects related to IPTV, web, mobile, or internet content distribution. The successful candidate will join a multi-disciplinary team of scientists dedicated to advance and use computational methods to solve challenging user-oriented problems.

The applicant should have a Ph.D. degree in Computer Science, Applied Mathematics, Statistics, or other related scientific disciplines, combined with strong computational modeling and/or algorithmic skills. Knowledge and experience in additional areas such as statistical data analysis, data mining, machine learning and pattern recognition, and other topics in artificial intelligence are desirable. Experience in the Recommender Systems field will be taken into account but it is not strictly necessary.

Although this particular position is designed for a postdoctoral candidate the group is also actively seeking for doctoral students in this area that might be interested in finishing their Thesis in Telefonica Research or in doing a research internship. We will also take into consideration strong candidates that are looking for a research position or senior research position. If you are in either of these situations please do not hesitate to apply.

We offer competitive salary and benefits and a great working atmosphere in beautiful Barcelona (Spain).

Screening of applications will begin immediately and continue until the position is filled. An initial appointment for a one year term is anticipated with the possibility of reappointment.

Inquiries and applications should be sent to

Xavier Amatriain

cc Nuria Oliver

with the subject line "RS Application"

Latest research on Recommender Systems

2008-09-23T03:45:00.000-07:00

In the past months I have focused my activity on Research in Recommender Systems. Although I have worked in many more projects so far we have 3 publications ready and undergoing a submission process. Although I cannot give any details until they are accepted I thought it would be interesting to post what they are about. Just to give you a feeling of the kind of work we are doing.

In "The Wisdom of the Few: Using Experts to Predict Ratings from the Crowds" - written with Neal Lathia (UCL) and Haewoon Kwak (Kaist) - we devise a variation over traditional collaborative filtering in which neighbors are sought from a database of "experts". We try to predict user opinions by simply using a very reduced number of expert opinions.

In "Collaborative Filtering With Adaptive Information Sources" - again with Neal Lathia but also with Josep M. Pujol from Telefonica Research - we study how the problem of collaborative filtering can be turned into a problem of data classification. Instead of using an ensemble of algorithms we propose to use an ensamble of data sources.

Finally in "I like it... I like it not: Evaluating User Ratings Noise in Recommender Systems" - with Josep M. Pujol and Nuria Oliver from Telefonica Research - we evaluate user natural variability when giving feedback through ratings. In our study we measure different effects and discuss how this might be related to the "magic barrier" in Recommender Systems.

More details soon... if they are accepted :-)

Models of R&D

2008-09-22T15:40:00.000-07:00

In a recent presentation I had to explain what the different models of industrial R&D are in IT. My goal was to explain the basics in a few line so I came up with this idea of presenting R&D organizational models in terms of 3 simple and distinct proto-models.

I call them the Microsoft model, the IBM model, and the Google model. However, before presenting them I need to stress a few disclaimers:

Overall what I am presenting here is a gruesome over-simplification. Reality is much more complex than this. However, I think these models sample the solution space pretty well. I think that any other R&D model can be understood as a variation or combination of these 3.
I use company names for the models because from what I know they are pretty good representations of what goes on in these companies on average. But again, things are more complex than this and in these very large companies there is space for everything and you can surely find counter-examples where everything works differently.
As you can read for instance here, large companies are in fact transitioning and describing their model is indeed a moving target.

The Microsoft model

Microsoft Research is a separate company from Microsoft
MS Research is entirely devoted to research
There are many more researchers than developers
There is an in-house techtransfer team
Techtransfer model

Project always started on the Researh side
Inhouse techtransfer takes upon interesting research project
Build prototype and convince product/marketing teams

The IBM model

IBM Research is also separated from Development (a la MS)
But, when a Research project is successful

The researchers are moved into the D team to help develop/coordinate

Extreme patent incentives not so many for publications
Constant feedback from Development to Research on what they need

The Google model

Google has many PhDs/Researchers

Many doing advanced development
Others doing applied Research
Very little incentive for publication/patent

Idea: research “embedded” into everyday product development
However: small research groups are being created with some particular focus

Results

It is hard to say which one (if any) is the best model. However some of the results, again oversimplifying, are clear:

The Microsoft model results in low techtransfer and little innovation coming from research into the products. However, they have high publication rates and happy researchers.
The IBM model produces moderate techtransfer and research-related innovation. They do have a high patent-rate but that has more to do with the quality of lawyers than the quality of research.
Google has a high techtransfer activity (everthing is techtransfer). However they have little publication activity so some might say they are not really doing research.

Again, this is a very much simplified picture of reality. But I would like to hear some feedback/comments on what you think.

Agile Research

2008-08-07T16:20:00.000-07:00

The 3 main features of any agile methodology are:

Iterative development
Focus on executable software
Adaptation to change
Focus on individuals

But what does all this have to do with Scientific research? My hypothesis, which I presented in a talk recently, is that research is inherently agile. Research teams can and should indeed apply many of the agile principles in their day to day work.

If we take a look at the traditional Scientific Method process (see figure below) we can see that it shares many features with agile methodologies. Above all, the scientific method is inherently iterative. In every iteration we set a hypothesis, test it, and analyze the results.

In this sense, it resembles most of Agile process definitions such as Scrum or XP: the first part of the process deals with setting up the general questions. This is similar to Scrum's sprint 0 in order to define the product backlog or to XP's Planning Game. Then we start doing iterations, or sprints, in which we ask smaller questions or decide which of the original hypothesis we are going to test.

Finally, it is interesting to note that the whole process is hypothesis-driven. This is very similar to the Test-driven development practice promoted by XP'ers. And there are many other practices (such as Peer programming or Simple Design, equivalent to Occam's Razor) that have a direct mapping between agile software development and Scientific Research.

Based on this I propose an agile scientific method in which:

Use an Iteration 0 (or XP planning game or Scrum Sprint 0) to build up general hypothesis and list those "stories" that you would like to have in your final article (optionally writing the article stub already).
Every 1-2 weeks come up with a list o prioritized taks

List all possible tasks (ideally 1-2 day workload)
Measure interest of task towards final goal
Measure cost in terms of predicted hours of work
List them in order of priority value (= interest - cost)
In the iteration planning re-evaluate general hypothesis.

Plan stories as an "executable hypothesis": if I can prove hypothesis H then it should happen that tests t1 and t2 should pass.
Maintain collection of tests as a record of experimetns.
Use the red-green-refactor cycle of Test-driven development to refine model while still complying with experimental data.

If you are interested you can read the full presentation that Gemma Hornos and I did at a recent seminar where she also presented some agile initiatives in Telefonica.

It's so hard to blog...

2008-07-27T15:46:00.000-07:00

when you get used to twitting!

I have around 6 unfinished posts and I find it really hard to get the time to polish them to a bloggable level. Investing that time in finishing a blog post is hard when you know that in the same time you could probably send 10 interesting microposts in twitter.

In any case, it's not that I like twitter that much. I like the idea of microblogging but now twitter is turning into everything else: post board, public mailing list... It is really hard to find interesting twitters and most of them are more and more using it as a social post board instead (I cannot exclude myself from this category either).

In any case, I hope to get a couple of this blog posts finished soon.

Standards for 3D Scene Description

2008-06-08T15:17:00.000-07:00

What can you do if you want to describe a 3D scene including audio and graphics using some kind of standard? This is the result of a quick survey we did for CLAM, feedback is welcomed!

There are several standards and open languages to describe 3D graphics, but what if you want to add 3D audio description into the scene? Well, that limits your choices but there are still some possibilities.

First, you can take a look at MPEG4's BIFS and AudioBIFS. BIFS is in fact an extension of VRML. MPEG4's SAOL (Structured Audio Orchestra Language) can also be included inside AudioBIFS. Unfortunately I don't know of any Open Source reference implementation of this, although parts of it have been implemented, for instance, in Ross Bencina's Audio Mulch.

X3D is a W3 standard that is also a sucessor of VRML (with the advantage of using an new XML format), in which the sound is integrated too. There are no complete implementations of X3D available and all of them are written in Java. X3D shares a lot with BIFS. As a matter of fact the MPEG standard already includes a link to X3D.

Another possibility is using VML from the Verse project. This project is very much related to Blender. Using VML can tie you into their data model. However Verse is released with a BSD license.

Finally, SpatDIF, is an extension of the SDIF format for sound interchange. It uses OSC for real-time communication and SDIF as the intermediate file format. There are no available implementations yet and it is unclear whether it can be easily extended to include 3D graphics. However, SDIF can be transcoded into SAOL and, as already mentioned, SAOL can be included inside MPEG4's BIFS.

So, in summary, it looks as there are several competing and complementary efforts but none is sufficiently mature yet. Again, any feedback on related experiences will be appreciated.

Agile Methodologies in Telefonica R&D

2008-05-26T22:58:00.000-07:00

Some time ago I started having interesting but informal conversations with a manager in Telefonica R&D who was also interested in Software Engineering and methodologies. My point was that in a company like ours where innovation is a key issue and almost all development is geared toward bleeding edge products, Agile methodologies should be used. Besides, development teams are usually small, requirements are not clear at the beginning, and there is a need for having working prototypes soon in the development cycle... there was no doubt.

After not much we managed to convince some key people, including our Methodologies Division (who had had bad experiences in the past with Agile).

Although I was much for starting with eXtreme Programming and do a bottom up approach it soon became obvious for them that in such a large organization as ours they needed a slightly more structured approach with a focus more on management and less on development. That is why we chose to go for Scrum.

Actually, what we are deploying is a mixture of Scrum for the management layer and xP practices on the development layer. This seems to be a favored approach nowadays in many companies (see here, here or here, for example).

Just a couple of months afterwards we have more than 10 projects that are successfully working Agile. And so far everything seems to be positive: developers feel better and even some of our clients are now turning to us for implementing agile methodologies in their companies. It is still a bit soon but things look bright!

Bottom line, be careful with informal conversations you have by the coffee machine at work... they might become true and end up having a huge impact :-)

ps. I hope I have some time to blog a bit more about how to combine xP and Scrum but I have many posts in the queue right now.

Digital Beers

2008-05-13T16:59:00.001-07:00

One of the coolest things of "being digital" (i.e. having a blog, twittering...) is how much of that digital activity ends up bleeding into your analog world. Every now and then people that know about me from the web get in touch and tell me that they are going to be around Barcelona. This way I have met really interesting people and projects.

So, if you are going to be around Barcelona (or any other place that I might be visiting) and want to chat while having a non-digital beer please feel free to get in touch. I am always up for that!

A Course on Recommender Systems

2008-04-20T16:27:00.000-07:00

Last week I gave an internal course on Recommender Systems in Telefonica. Although I only had 12 hours I ended up preparing a course that could well expand over at least 24 hours of class. Except from some details I am pretty happy with the syllabus I came up with. Just in case it might be of any help to anyone, this is the index of the course:

PART I. Introduction to Recommender Systems

The paradox of choice
What is a Recommender System?

The recommender problem
General scheme of a RS
Tools of the trade

Approaches to Recommendation

Collaborative Filtering

User-based
Item-based

Memory-based
Content-based
Other approaches

Demographic Methods
Utility Methods
Knowledge-based

Hybrid approaches

Weighted
Switching
Mixed
Feature Combination
Cascade
Feature Augmentation

Evaluating RS
Personalized Search

Part II. Data mining for RS

Introduction

Why mine data?
Data mining tasks

Data Preprocessing

Types of data
Problems with data
Aggregation
Sampling
Reducing dimensionality (SVD)
Feature Selection
Discretization and Binarization
Variable Transformation
Feature Selection

Distance Measures
Classification

General Approach
Decision Trees
Rule-based
Nearest-Neighbor
Bayesian Classifiers
Artificial Neural Networks
Support Vector Machines
Ensambles of classifyiers
Issues in classifyiers

Model Overfitting

Evaluation of Classifiers
Comparing Classifiers
Metrics for classifyiers

Cluster Analysis

Introduction
K-means
DBSCAN
Cluster Validation

Association Analysis

Frequent Itemset Generation and the Apriori Principle
Rule Generation

PART III. Designing a RS

Defining the problem
Working with the data
Taking context into account
The decision process
Presenting results
Some notes on domain-specific adaptation

The Future of Web Search Workshop

2008-04-01T15:11:00.000-07:00

Yahoo! Researh Barcelona and the UPF are organizing a very interesting two-day workshop in Andorra starting on Thursday. The workshop is the third of a series that, under the same name of "Future of Web Search", started in 2006. I will be giving a talk entitled "Search and Recommendation: two sides of the same coin?". Below is the abstract that can give you an idea of what I will be talking about:

Recently the field of Recommender Systems has gained growing popularity among the research community with new conferences such as the ACM Recsys going into its 2ond edition and established conferences such as SIGKDD or SIGCHI focusing a great deal of attention on this topic.

The Recommendation field started from a different background than web search, namely Data Mining and HCI versus Information Retrieval. While the goal of Recommendation Systems is to optimize a fitness function between content and users by "discovering" hidden relations in the data, Search Engines focus on "retrieving" pre-existing data.

However there are clear trends that point to both fields coming closer together. On the one hand, web search is becoming more and more personalized, highlighting the need for user profiling and collaborative filtering. On the other hand, it is becoming clear that in many cases search strategies are essential for the performance of Recommender Systems.

As a result, some claim that search is just a "simpler form of recommendation", where the fitness function to be optimized is that of a generic average user (e.g. using algorithms such as Page Rank) Obviously statements in the opposite direction can also be made. In this talk we will assume that the audience is familiar with Web Search systems and therefore we will focus on describing the basic techniques and current research trends in Recommender Systems, highlighting where and how they are similar or different. At the end of the talk, We hope to convey the message that the "Future of Web Search is in Recommendation", hoping that such a claim will spark an interesting discussion and debate throughout the workshop.

Here you can see the detailed workshop program with very interesting speakers including Yahoo's own CDO, Usama Fayyad.

Call for Students: CLAM in Google Summer of Code 08

2008-03-23T17:53:00.000-07:00

(Please help distribute)

We are glad to announce that 2008 summer is also going to be a Summer of Code for CLAM . In other words, CLAM has been accepted as a mentoring organization for the Google Summer of Code, a program that offers student developers stipends of 4500 USD to write code for open source projects.

CLAM (C++ Library for Audio and Music) is a project that aims at developing a full-featured application framework for Audio and Music Applications. It offers a conceptual metamodel as well as many different tools for that particular domain. One of its most relevant features is the availability of a visual building dataflow application that allows to develop rapid prototypes without writing code. The project started 7 years ago and, among other highlights, it won the ACM award to the Best Open Source Multimedia Software in 2006.

Now we are looking for smart students who enjoy coding free software so that they can earn some bucks for the summer. Last year, GSoC 2007 was a very fun and productive experience and we are looking forward to repeat it. Take a look at the CLAM GSoC 2008 wiki page for more information on how to apply and some sample ideas for projects.

We are waiting for you!

Application deadline: March 31

If you have any question about any of the information below please
contact clam-info@iua.upf.edu or join the #clam channel at FreeNode
IRC.

CLAM 1.2 released

2008-02-18T14:03:00.000-08:00

Many things have happened in between our last two releases but we finally managed to pull the 1.2 release together codenamed "the gsocked plugged in release". This release includes all the cool stuff our students from the Google Summer of Code developed. For this reason CLAM was also featured in the GSoC blog.

Congratulations to everyone who worked on this release and special thanks to David who did a great job as release manager for this one.

More news and downloads in CLAM website.

Everything is a Graph (part 2)

2008-02-12T14:09:00.000-08:00

In a previous post I talked about how graphical models of computation are being used beyond the "traditional" areas of networking and low-level system modeling. It may come as a surprise that being this such a rich an useful paradigm it has not become so widely spread as the object-oriented approach. So now I will discuss whether that assessment is true and what might be the reasons.

Why don't we have graph-oriented programming languages?

First of all it is interesting to note that object-oriented itself comes from graph-oriented (or process-oriented) approaches. It is widely accepted that Kristen Nygaard's Simula language was the first OO language to see the light. However, Simula was a (as its name might imply) a simulation language in which the most important concept were the processes. Simula did follow a graph-oriented approach and it was only in their later versions (Simula 67 and later) that the idea of "objects" was explicitly presented. So, in some sense, OO can be seen as a generalization of the graph-oriented approach. As a matter of fact you can indeed understand a graph as a set of interconnected objects called nodes. In the same sense you can read an OO design in a graphical way where classes or objects are nodes and relations become the graph edges. You can read more about this interpretation of OO in my thesis.

Furthermore, at some point in time a few people realized that it would make sense to define a graph-oriented paradigm and design languages to support it. This gave birth to the so called actor-oriented and process-oriented languages. I am unsure why these languages miserably failed to make it mainstream but I can safely assume that it was a mix of different factors such as bad PR (who would chose an approach called actor-oriented?), bad implementation, and even bad timing.

However, does that mean that graph-oriented languages have failed as a whole? And my answer to this would be a definite NO. What happens is that the graph-oriented paradigm lends itself much better to a graphical (for based on graphics) a representation. Therefore, graph-oriented languages skipped on or two steps in the logical evolution of a programming language: general purpose textual language -> general purpose graphical notation -> domain specific graphical models. The OO paradigm started producing a large collection of textual languages, then a general purpose graphical notation (UML), and is currently gearing toward domain-specific graphical modeling languages.

Curiously enough graph-based models jumped directly to the latter and you can find many examples of domain-specific graphical languages that go from some with a broader scope such as Simulink or Ptolemy to some that target a more specific domain such as CLAM, Pd, or GStreamer.

And now with OO tending to the same place and tools like Metaedit offering ways to quickly develop your graphical DSM we are seeing how the OO and graph-oriented paradigms are finally coming together again.

So yes, everything is still an object... and almost everything is becoming a graph!

Pay-what-you-listen

2008-01-25T16:11:00.000-08:00

It occurred to me that with all these services (such as lastfm, mystrands...) tracking what you listen during the day we can finally implement a really fair business model for music: pay-what-you-listen. Imagine you paid a fixed subscription price and you were ensured that this money would go directly to the artist and divided according to how many tracks from that artist you played during that month. I'd sign in for this service today!!

Why opening APIs is not enough: the Facebook vs. Lastfm case

2008-01-24T13:41:00.000-08:00

Every day I read of more people that are becoming deceived with Facebook and its usuability... I am one of them. The other day I had a depressive experience when I tried to find an interesting message someone had sent over the past few weeks. I could not find the message. Was it a wall posting? or something SuperWall or FunWall? Or maybe simply a note... or a link. Or a message or anything sent with any of the dozens of applications that I have to have.

Facebook is ok to stay tuned to what your friends are up to. The problem is that in order to do that you are forced to accept the many applications that people end up aggregating. And then it is a complete mess. You have many friends with many different apps and organizing information in a sensible way is impossible so you end up reading things you don't care about and possibly missing interesting information. Opening the API made Facebook big but, will it die of success? Opening things up and hoping that they will organize themselves works in some cases (e.g. the web) but in some others is a recipe for disaster.

Plaxo does a much better job of organizing things and keeping them manageable. However plaso is still too new to guess in which direction it will grow.

Lately I have been becoming a more intensive lastfm user. And I can say that it is becoming my favorite social network. Of course I am an absolute music lover and that makes a difference in this case. But that is not the point. The point is that lastfm is a "focused" and manageable app. You sign in because of music and find friends because of music but you can also add your pre-existing friends and laugh at their bad taste :-) Also lastfm has been constantly adding new features like the recently announced availability of full tracks for free.

You can argue that lastfm is simply a tiny part of what facebook is. So what? It is useful, enjoyable and fun, what else can you ask for? Completeness? If you think so I'd recommend you read Barry Schwarzt's The Paradox of Choice: Why More is Less, or watch his great talk at Google.

So instead of a dominating social network like FB aggregating everything I envision dedicated ones (such as lastfm, flixter, mystrands....) becoming more and more popular and aggregating services like Plaxo being used as a common entry into all these different worlds.

In a sense is a bit like the evolution that took place in Software Framework design. At one point people were trying to build the "one framework for all". Even if that was ever possible, users of such a monster would be unable to understand and use it. The same has happened to all-encompassing metadata standards such as MPEG7, ontologies...

Bottom-up design has always been better than top down approaches and I believe Social Networks will prove no different in that sense.

Telefonica Research: Doctoral Researcher position in Recommender Systems

2008-01-22T09:23:00.000-08:00

(Thought I'd pass this on as it clearly involves me :-)

The research group on Internet in Telefónica R&D Barcelona invites for applications for a Junior Research position in the area of Recommendation Systems.

We are looking for dynamic, creative, and resourceful individuals to join our research efforts in modeling of complex systems and networks related to recommending engines. Our research impacts all areas of the company, including projects related to IPTV or internet content distribution. The successful candidate will join a multi-disciplinary team of scientists dedicated to advance and use computational methods to solve challenging user-oriented problems.

The applicant should have a Master degree in Computer Science, Electrical Engineering, Applied Mathematics, Statistics, or other related scientific disciplines, combined with strong computational modeling and/or algorithmic skills. Knowledge and experience in additional areas such as statistical data analysis, data mining, signal processing, machine learning and pattern recognition, and other topics in artificial intelligence are desirable.

The candidate will carry out both theoretical and applied research leading to a PhD degree in Computer Science and will actively participate in innovative projects related to this area of research in the company. Our research group follows an open research model in collaboration with universities and other research institutions and favor the dissemination of our work both through publications and technology transfer. The successful candidate will also be enrolled in a local university and will be tutored by a Professor in order to obtain the PhD degree. Otherwise particular agreements with the candidate's
original institution are also feasible.

Although this particular position is designed for a doctoral student candidate the group is also actively seeking for postdoctoral candidates in this area. If you are in this situation please do not hesitate to apply.

We offer competitive salary and benefits and a great working atmosphere
in beautiful Barcelona (Spain).

Screening of applications will begin immediately and continue until the position is filled. An initial appointment for a two years term is anticipated with the possibility of reappointment.

Inquiries and applications should be sent to

Xavier Amatriain <xar@tid.es>

with the subject line "RSDoc Application"

* Telefónica is a world leader in the telecommunication sector, with presence in Europe, Africa and Latin America. As of March 2007, Telefónica had 206.6 million customers.

Telefónica Research and Development is the innovation company of the Telefónica Group. Owned 100% by Telefónica, this subsidiary was formed it 1988, with the aim of strengthening the Group's competitiveness through technological innovation.

It is the most important private R&D company in Spain, in terms of both activities and resources, and in terms of number of staff, and it is one of the most important companies on the continent as regards participation in European Research projects.

Everything is a graph (part 1)

2008-01-20T14:24:00.000-08:00

Alan Kay summarized object-orientation by stating that "everything is an object". Therefore any part of the world that needs to be modeled in a software system can be described in terms of objects and their relations.

Already in my thesis I worked on relating the object-oriented paradigm to graphical (or graph-based) models of computation in the context of signal processing systems. Actually there are many graph-based frameworks and applications in the context of signal processing, multimedia, and related fields. However, lately I have been working with graphical models in many different situations.

Graphical models are gaining more importance in data mining, for instance, through the use of bayesian belief networks and other graphical models. On the other hand the study of complex networks and systems has introduced yet other ways to look at graphs from a statistic perspective.

So in a sense, graphs can give an us an equivalent yet complementary view to the object-oriented paradigm. Where in OO we have objects in graphs we have nodes, and where in OO we have "relations between objects" in graphical models we have edges. So we can conclude that Everything is a Graph.

The Future of the Music Business

2008-01-02T13:39:00.000-08:00

I just read a couple of very interesting articles in Wired Magazine discussing the Future of the Music Business. In the first one, David Byrne, concludes that although the "traditional" model for music business is close to over music is much more than that and the future is much more promising for artists who can now chose from up to six different models. In the second one Byrne has an interesting conversation with Radiohead's Thom Yorke.

Very interesting reads from two of the most interesting minds in the business, indeed.

Cool Projects in Telefonica R&D: the New Generation TV

2007-12-29T15:22:00.000-08:00

In this new series I will introduce some of the cool projects we are working in Telefonica R&D Barcelona. Although I am not working full-time in any of these projects I do have some contact with them and the people in charge.

The first project I decided to talk about is one of the most eye-catching projects we have. No wonder it usually becomes the center of attraction of the many demo and open show cases we have in our center.

But before it is important to explain that Telefonica's business not only deals with fixed and mobile phone. For instance, Telefonica owns Imagenio, an IPTV over ADSL that operates in Spain. As a matter of fact, Imagenio is one of the world's largest operating IPTV networks in the world. Imagenio is streamed directly over standard ADSL with a QoS and it has proved that this technology is scalable enough to serve large audiences and offer new interactive services like video-on-demand that allows the user to watch the content any time in the following 24 hours and stop/play/rewind at will.

The project that I am writing about is actually a number of different projects, all of them working towards defining the new generation of interactive TV. On the one hand we are working on improving the audiovisual experience by providing 3D image on autostereoscopic displays and 3D audio. The first pilot is about to be deployed in the following months. On the other hand, there is ongoing research and development on providing personalized content and advertisements by creating user profiles and deploying automatic recommendation systems.

More innovative and breakthrough ideas are also on the works but I will keep those to ourselves for the time being for obvious reasons :-)

Imagine 2040: a Christmas Story

2007-12-22T15:49:00.000-08:00

Last week we had a story writing contest at work. Of course the context was a bit geeky so the story had to be about imagining life in 2040. If you are curious here you have my original submission in spanish and a quick translation into english.

University Research Groups trying to become companies: the MTG/BMAT case

2007-12-20T15:13:00.000-08:00

When some weeks ago I received an email about some researchers being fired from the Pompeu Fabra University the issue smelled really bad but I did not plan on writing about it on my blog. However, after talking to some of them and realizing that they have received no support from the University or their colleagues I feel that I least have the obligation of making this public.

The five researchers that I mentioned had previously sent an internal letter complaining of the lack of separation between the Music Technology Group, a research group at a public university, and the BMAT company. As a response they were fired from the group and the university and the email one of them then sent to the Chancellor of the university has so far gone unanswered.

This is the last of many sad episodes related to the creation of this pseudo-company. Among these, the one that has affected me personally most: the mistreatment of the CLAM project. Even though CLAM was successful enough to win an ACM award or get into the Google Summer of Code, for instance, some time ago it was decided that the project was not only not interesting for the MTG group but was actually "dangerous". The people in charge of BMAT/MTG (it's really hard to distinguish which is which) interpreted that it was a direct competition for some of the projects in BMAT. As a result we had to move CLAM, and its two remaining developers, onto a different university group.

Maybe what these five fired researchers did not know, being all of them non-Spanish, is that these research groups that are half companies and vice versa are common practice here in Spain and I would dare to say in many other places in Europe.

Let me be clear: I am all in favor of spin-offs and start-ups connected to universities. This should be done by supporting researchers in the institution that feel can create a company by commercializing a particular idea or application that was originally developed in the university. However a very different thing is to give all the public IP related to a public institution to a company (see here how BMAT has an exclusive license for all MTG's technologies). Even worse it is to hire researchers in the public institution, with public funds, who are in fact working for the company and only respond to the strategy the company dictates.

But, to be fair, I cannot blame the people behind BMAT for acting like this. As a matter of fact they have mostly been pushed to doing (or at least allowed to do) what they are doing because the University system in Spain is, as its whole education system, extremely mismanaged. Full-time professors have low salaries (so imagine doctoral or postdoctoral researchers) and the overall investment in R&D is to laugh at (or to cry), see here and here. In this situation, they have no other way out that to complement their salaries by getting involved in mostly European Research projects or to create companies that are mostly subsidized. Furthermore, academics are then mostly evaluated on how much money they have brought into the institution, in one way or another.

So there is no easy way out of this as it involves both governments and companies investing more in R&D. But the least one can expect is for people behind these initiatives and the university to behave under some moral principles. And firing five people because they complain of an unfair situation does not fit under my criteria of morally acceptable behavior.

Sorting rows in text files

2007-12-07T14:19:00.000-08:00

If you work with scientific data in text files you have probably faced the need to sort rows in a text file following some criteria. If the text file is small enough you can import it as a CSV into a spreadsheet editor and sort it there. But if you have thousands of rows you have to use a script or command line instruction such as *sort* in Linux.

The problem with sort is that it is not easy to get exactly what you want ... and that is what happened to me some days back. I had a file that included some rows like the following:

1 100 9
1 17 8

I wanted to sort rows by first column and then second so I tried using commands such as:

sort -n -k 1,2

Unfortunately this did not work out as it would leave the sample lines in the exact same order. Supposedly the -n switch should make sort interpret columns as numbers but that is not exactly the case. According to sort 17 is greater than 100 simply because it is a 1 followed by a 7, versus a 1 followed by a zero.
Using all other switches did not help much.

So, in order to get this working I had no other option than to pad with left zeros to get equal length numbers in all columns. For doing that I used awk with its printf functionalities.

Putting that together in a one-liner yielded the following command that worked like a charm:

awk '{printf "%4.0f\t 005.0f \t %1.1f\n",$1,$2,$3}' data.txt
| sort -n -k 1,2 > test

Open Research Day at Telefonica R&D

2007-12-02T14:55:00.000-08:00

Last week we had our first Open Research Day in our newly created research group in Telefonica R&D. It was a great event that had invited speakers from external labs as well as our own presentations and demos. Also we had a pretty good attendance of researchers from mostly local companies and universities.

The event had also some impact in spanish press as you can read in Expansión, El Pais, or TV3.

I gave a presentation talking about some of our projects in the area of Recommender Systems, including some of the things we are doing in relation to the Netflix prize. You can download the presentation here.