Three new research papers (for TPDL’13) about user demographics and recommender evaluations, sponsored recommendations, and recommender persistance

After three demo-papers were accepted for JCDL 2013, we just received notice that another three posters were accepted for presentation at TPDL 2013 on Malta in September 2013. They cover some novel aspects of recommender systems relating to re-showing recommendations multiple times, considering user demographics when evaluating recommender systems, and investigating the effect of labelling recommendations. However, you can read the papers yourself, as we publish them as pre-print:

Paper 1: The Impact of Users’ Demographics (Age and Gender) and other Characteristics on Evaluating Recommender Systems (Download PDF | Doc)

In this paper we show the importance of considering demographics and other user characteristics when evaluating (research paper) recommender systems. We analyzed 37,572 recommendations delivered to 1,028 users and found that elderly users clicked more often on recommendations than younger ones. For instance, users with an age between 20 and 24 achieved click-through rates (CTR) of 2.73% on average while CTR for users between 50 and 54 was 9.26%. Gender only had a marginal impact (CTR males 6.88%; females 6.67%) but other user characteristics such as whether a user was registered (CTR: 6.95%) or not (4.97%) had a strong impact. Due to the results we argue that future research articles on recommender systems should report demographic data to make results better comparable.

Paper 2: Persistence in Recommender Systems: Giving the Same Recommendations to the Same Users Multiple Times (Download PDF | Doc)

How do click rates vary between research paper recommendations that were previously shown to the same user and recommendations shown for the very first time? To answer this question we analyzed 31,942 research paper recommendation given to 1,155 students and researchers. Results indicate that recommendations should only be displayed once and click-through rates for ‘fresh’, i.e. previously unknown, recommendations are twice as high as for already known recommendations. However, results also show that some users are ‘oblivious’. It frequently happened that users clicked on recommendations they already knew. In one case the same recommendation was shown six times to the same user and the user clicked on it each time again. Overall, around 50% of clicks on reshown recommendations were such ‘oblivious-clicks’.

Paper 3: Sponsored vs. Organic (Research Paper) Recommendations and the Impact of Labeling (Download PDF | Doc)

In this paper we show that organic recommendations are preferred over commercial recommendations even when they recommend the same freely downloadable research papers. Simply the fact that users perceive recommendations as commercial decreased their willingness to accept them. It is further shown that the exact labeling of recommendations matters. For instance, recommendations labeled as ‘advertisement’ performed worse than those labeled as ‘sponsored’. Similarly, recommendations labeled as ‘Free Research Papers’ performed better than those labeled as ‘Research Papers’. However, whatever the differences between the labels were – the best performing recommendations were those with no label at all.

Deadline for submitting the camera-ready papers is Sunday, June 9th. If you can spare some time, please feel free to read the papers and let us know any spelling errors, typos, grammar problems, etc. We will gladly incorporate your suggestions into the final version.