I think the comments helped me to clarify my thinking a bit.

I think in this follow up post you’ve actually made a more sophisticated and science specific re-statement of Shirky’s case at least as I (thought I) understood it. Where you are differing is on the consequences of his conclusion if I’m reading you and him right? He sees it mainly as a technical problem to be solved (build better filters) whereas you are concerned about the consequences of applying strong er filters

You’re generally on the right track here. I think the big difference is that someone like Shirky would assume that better filters would solve the problem, whereas I don’t agree. If the problem is defined as an inability to take in the necessary information, and more filtering means throwing out vital information, then no, it’s not a solution. That’s more of a surrender than a solution.

1. Publish less (simply reduce the amount of science being done until we can cope)
2. Publish smarter – by which I mean make the medium appropriate for the information that people need to get at so they can get at it more efficiently. I’m thinking mainly of mechanisms for e.g. data publication here – I don’t need to read a paper if all I want is a single number, or a single data file, or the details of a method.

I think these two are intertwined in some ways. Publishing less is probably not a practical option, given the business models of the vast and expanding number of journals out there (they have to publish something in order to exist), and the need for an expanding number of scientists to have concrete markers of their research achievements (to push for graduation, tenure, grants, etc). Limiting the number of publications would hurt the open access/reader pays type journals the most–numbers so far show that they need to publish higher quantities of work with less editorial overhead in order to be financially viable (although for papers that are just indexed data as you suggest, open access would be a great fit).

The semantic technological improvements many are predicting would also hopefully create efficiencies in taking in the literature.

3. Filter harder and finer – become more specialized – and I agree this has potentially bad consequences – or at least would require changes in the way the research effort is managed.

Probably my main point here–more filtering means less-rounded knowledge. Some of the most interesting science is done cross-disciplinary, and lessons learned in one area can be highly valuable in another. Cutting yourself off is a bad move.

4. Distribute the reading process amongst a trusted network and re-aggregate the information in summarised form. Do you really need to personally read all of those papers? Could you get away with an executive summary for, say 50% of them?

While I’d agree, the problem is that you can’t know in advance which 50% prove to be important. Sometimes you won’t know for years. When you read paper B, you put that together with the results of paper A that you read 2 years ago and you’re onto something new and unique. The devil is often in the details, and what one scientist reads into a paper is very different from what another reads. With a background in developmental biology, I would read something very different from the first GFP papers (a vital dye for studying cell migration) than would someone like Roger Tsien (a means of measuring protein:protein interactions via FRET). One key to your comment is the phrase “trusted network”. We need to think more in terms of the groups where scientists work, labs, collaborators and departments, than in the free-for-all worldwide sorts of networks so many are coming up with. If you’re working on that level, online exchanges of information aren’t a big step beyond what scientists already do with lab meetings, journal clubs, etc. They’re just a different presentation method, albeit perhaps a more efficient one. There’s a freedom to speak openly and critically in those small private groups that is lost when they become open public forums. It’s unclear if that small gain in efficiency though, is enough to overcome the current expansion of the literature.

There’s also the problem of the time needed to write those executive summaries taking away from either doing more research or reading more papers. How one balances the altruism for the group/community versus the need to advance one’s own career is going to be a problem for a very long time.

There’s a real limit though, in how much you want to turn over to have the group do for you. I think what many proponents of Web 2.0 for science are underestimating is how much of the nature of science is individual effort. The great breakthroughs, the creative leaps, come from individual minds, not from the hive mind of executive summaries. There’s a level that no matter how much you want to automate things and distribute things, you still need to put in deep individual intellectual effort to fully understand anything and then to creatively move beyond that. It can’t be avoided, it can’t be fobbed off on others.

Any researcher has a limited amount of time to read papers – and we can for convenience sake agree that for many researchers, even for the papers they have to be well acquainted with, we have reached saturation. So we either have to give up, or improve filters. It seems there are really only four options:

1. Publish less (simply reduce the amount of science being done until we can cope)

2. Publish smarter – by which I mean make the medium appropriate for the information that people need to get at so they can get at it more efficiently. I’m thinking mainly of mechanisms for e.g. data publication here – I don’t need to read a paper if all I want is a single number, or a single data file, or the details of a method.

3. Filter harder and finer – become more specialized – and I agree this has potentially bad consequences – or at least would require changes in the way the research effort is managed.

4. Distribute the reading process amongst a trusted network and re-aggregate the information in summarised form. Do you really need to personally read all of those papers? Could you get away with an executive summary for, say 50% of them?

I still feel you’re conflating volume and quality of research.

While they’re not the same thing, there is a relationship between the two. If there are only 10 labs in the world doing research on subject X, and only 2 of them produce quality results, must that number stay static if the field expands and there are now 10,000 labs working in that field? Is it possible that more than 2 labs could be doing quality work? What about all the students and postdocs who came out of those 2 labs over the years? Could they add to the volume of high quality research being published? Or is there an absolutely constant limited number of quality results that can be produced, regardless of the size of a field?

My argument would be that a small percentage of researchers in any given field produce high quality results. As the field increases, that percentage may remain constant, or even decrease somewhat, but that still results in a larger quantity of good results produced. Which means more to read.

Here, I’m going to contend that the amount of high quality (=essential) research has not increased significantly since Aristotle’s day. Arguably, it has declined.

Wow. I have to vehemently disagree with that statement. That implies that there is only a constant, limited number of people on earth capable of doing quality research, regardless of the size of the population. That seems ridiculous. Let me put it to you this way–when Nobel Prize winners Andrew Fire and Craig Mello first started working on RNAi, it was easy for them to keep up with all the research on RNAi–there were very few people working on this unproven phenomenon. Once RNAi was established as an incredibly useful technique and interesting biological process, the number of labs working on it increased exponentially. Are you saying that even with this increase, the number of quality results being derived in RNAi research remains exactly the same as it was before the field was fully established? Or are there more quality results to track?

How many “me too” papers do you need to read? How many mutations in how many amino acids do you need to spend half an hour reading about (as opposed to pulling the information from a database)?

Those are the papers you were already filtering out. Now what do you do with the stack of “non-me too” papers? Again, if you really believe there’s the exact same amount of good science being done today as there was 30 years ago, there’s no point in continuing this conversation. If you accept that more researchers means more total results, and a similar percentage as always are going to be of high quality, then that means a greater amount to assimilate.