From December 2020 to January 2021, we asked you to participate in an online survey in order to help us understand how researchers are using dblp, and how dblp and its features are perceived by the public. The response exceeded our expectations: We received the amazingly high number of 1046 responses to our survey in total, with 760 surveys fully answered and 286 surveys partially completed. This response provided us with a plethora of helpful remarks, constructive criticisms, novel ideas, and feature requests that will guide us in the development of dblp in the upcoming years. We would like to express our heartfelt gratitude for your kind support of dblp!

In this blog post, we will share with you the aggregate results of the user survey (with figures based only on the 760 completed surveys), and a summary of what we have learned from it so far.

So, who is using dblp?

We asked you about your professional background. Probably unsurprisingly, the vast majority (93.4%) of users come from an academic background. But we also have significantly large groups of users with a background in industry (14.5%) and education (20.9%).

How would you describe your professional background? (multiple answers possible)

dblp is around for more than 25 years now, and many veterans of the computer science community know about and are using it. In this regard, we were happy to learn that we still find many junior researchers among our users, as 43% of you reported that you consider yourself to still be in an early stage of your career.

How would you best describe your professional seniority?

And since when have you been using our services? It turns out that almost half of you (48.7%) have been using dblp for more than 10 years! That’s an amazing figure, and we are grateful and happy for your continued trust in us. At the other extreme, we learned that only less than 5% of our users just started to work with dblp during the past year.

How long have you been using dblp?

How are people using dblp?

We already asked about your use cases with dblp in an earlier survey back in 2016. Based on that experience, we compiled a list of “standard” use cases and asked how often you are using dblp according to those standard use cases. The most frequent use case reported in the survey is the case of looking up a researcher’s bibliography in order to learn about their research output. More than 57% of all respondents stated that they are using this central feature of dblp at least once a week, while more than 85% stated that they are using it at least once a month.

In total, we found almost all listed use cases in regular use by around 70% of our users. The only true outliers were our open data APIs and the dump download options, which we found in regular use by only 24% to 30% of our users. We are well aware that these specialized features are only relevant to a smaller group of “power users”, so this number is not surprising. Nevertheless, survey comments from this group of users left no doubt that they greatly rely on the easy and open access to our data for its reuse.

I use dblp …

In addition to the standard use cases above, the survey also yielded a number of further types of uses as free text responses. The most frequently reported further uses include (in no particular order) using dblp …

• as a data source for curating your own personal bibliography
• to study coauthor relationships
• to study venue-author relationships
• to find versions of a given paper (e.g., preprints, journal versions, etc.)
• as a resource in hiring decisions or career committees
• as a resource in editorial or program committee work, e.g., when looking for reviewers
• to conduct scientometric studies
• to assess the legitimacy of a conference or a journal

What are users thinking of dblp?

In the development of dblp and the curation of its data, we aim for a number of different goals. For instance, we seek to disambiguate homonymous authors and to cover all of the most relevant publication venues in computer science. That’s why we asked you for your assessment of whether or not we are successful at it. We were happy to learn that the vast majority of respondents fully agree or tend to agree that we are achieving these goals. We were particularly happy to learn that more than 96% of respondents agree or tend to agree that dblp meets the needs of the computer science research community. The aspect that is perceived most critically is our efforts to disambiguate homonymous authors. Here more than 12% of opinionated responses disagreed with the notion that we are successful at this.

Do you agree or disagree with the following statements?

We also asked our users to directly compare dblp with a number of other research information services. We asked them to give a simple assessment whether dblp or the other service is considered more useful, or both services are considered equally useful. Of course, it made us quite proud to learn that dblp was assessed favorably when compared with almost all other major services.

The one exception here is Google Scholar, with the majority of respondents considering dblp and Google Scholar to be equally useful. Many users did let us know in their comments that they don’t even consider dblp and Google Scholar to be competing services. Instead, they consider them to be different tools which have their unique advantages with respect to different tasks.

How do you rate dblp compared to these other research information tools?

However, one should keep in mind that, unlike dblp, those other services above cover all science disciplines and do not focus only on computer science.

How satisfied are users with dblp’s features?

The dblp website and our open data service are the main interfaces users work with. Hence, it is important for us to know whether you are satisfied with our current implementation, and where we need to improve. It turned out that, in general, the survey participants are quite satisfied. The one noticeable outlier here are the “references & citations” detail pages, as more than 24% of respondents are rather dissatisfied or very dissatisfied with our implementation. This dissatisfaction we can fully understand, as this feature is still a work in progress and the availability of open citation data is still limited.

Please rate your satisfaction with the current implementation of the following features in dblp

Just as we had hoped, you had plenty to say when it came to suggestions for improving and expanding dblp. We compiled an internal list with literally dozens of your ideas for evaluation. Some of them have already been on our own to-do list, others were new and we are now seriously considering implementing a fair number of them. Frequent suggestions for improvements that we already had been considering or working on (some of which are actually quite close to a release by now) include:

• adding visualizations, like coauthor networks or topic clouds
• improved search capabilities and support for complex queries
• institutional bibliographies and search/filter by affiliation
• RSS update feeds for conferences and journals
• dblp as a knowledge graph and daily/persistent RDF dumps
• better dblp pages on mobile devices
• better interface/ticketing for dblp inquiries

Suggestions for new features which we would like to consider at some point in the future include:

• customization of default filter settings for bibliographies
• a way to create team/lab group bibliographies
• overview of trending topics and topical RSS feeds
• help pages and tutorials for dblp’s features

There have also been some frequent suggestions which we will most likely not pick up anytime soon. These include:

• author indices, venue rankings, or any other kind of scientometrics. We actually receive a lot of requests throughout a year asking us to add all different kinds of indices, rankings, or quality labels to dblp. But we as a team are just not convinced that we could do this delicate issue justice, or that this kind of number crunching does even more good than harm in general. So, of course, you are still welcome to use all of our curated data to build any index or ranking at your heart’s content upon it. But we would rather choose not to do so ourselves.
• adding abstracts to dblp. While we would love to do this, abstracts are creative works protected by copyright. So, in most cases we do not have the right to republish them. And even if we do, as is the case for many open access publications, it is still unclear how we can sustainably harvest abstracts in large numbers with clear and reliable license information attached to them. For instance, currently only 3.5% of all publications in dblp with a DOI provide both an abstract and license information via the Crossref API. That said, there is an Initiative for Open Abstracts (I4OA) which dblp supports. So, maybe there is hope that this will change in the future?

What we’ve learned and what’s next

Here are some further key leanings from the user survey:

• The dblp community is awesome. We asked for your opinions, criticisms, and suggestions, and you gave us plenty of those. Thanks again for that. But you did not stop there: 260 of you (that is 32.4% of all respondents) even took the time and used the free text fields to send us thanks, encouragements, and generally nice comments about our work, with some of them even being multiple paragraphs long. Thank you so much for that! Being blessed with such a concentrated stream of positive feedback did put a smile on our faces that lasted for days. And those comments will also be immensely helpful when we have to report to our public funders.
• It makes perfect sense to ask the community for their ideas. Sometimes it is hard to understand why certain ideas cannot be realized in dblp unless you are familiar with the details of dblp’s implementation. Because of that, we were a bit afraid that such a user survey would yield many reasonable suggestions that we just could not possibly follow. As it turns out, this fear was completely unwarranted. Quite the opposite, the responses made us realize that there are many potential improvements which are not that far out of reach and which promise to provide a positive impact on the user experience and the value of our open data.
• We need to better communicate our features. One thing that became apparent when reading your comments was that a fair number of dblp’s features, particularly those that have been added in recent years, are not generally known to our users yet. We received many comments stating: “I just learned from this survey that I can do THAT with dblp.” Such comments were made particularly often with respect to the integration of open citation data. The “addition of references and citations to dblp” ended up to be the second most requested feature even though it has already been implemented since 2019. We should definitely spend more time making our features better known and on documenting and giving guidance on how to use them. Many of our F.A.Q. pages are outdated, and we need to do something about that, too.
• We should do this again. We received so much valuable feedback and and so many insights to make it totally worth spending the time to create the survey and digging through the answers. While repeating a general survey will only make sense in a few years from now, it might still be worth exploring the idea of conducting a bit more regularly a number of smaller, more targeted questionnaires that address concrete topics.