I'd say the experiment (relying on the summary provided here) shows what works: - Papers that are obviously poor are rejected. - Papers that are uncontroversially good are accepted.

From a CS point of view, the problem is that in many communities, what is 'uncontroversially good' depends on subtleties only known to insiders who are willing to comply with the quirks of a community. And most communities are very defensive when it comes to shielding their top venues from relative outsiders, who are not sufficiently close to the inner circle of key players. As a consequence, getting the top-venue stamp (required to please the administration) becomes a social game. I think having a more open review process and stronger editorial control (vs. reviewer power) can mitigate this problem to some extent, for the following reasons: 1. When a paper is good, but upsets the subjective feelings of reviewers, the system should be implemented in a way that there is a high social cost for killing a paper because of subjective/cosmetic issues. Having venues where papers that are sent out to reviewers are most likely reasonable and should only be killed because of 'hard' flaws creates such a system. 2. When a paper is bad and the editors push it through, publishing details of the paper's editorial flow would impose social costs on the editors. (The reviewers, who may be potentially early in their career and may need protection, may even remain anonymous, the editors, who typically enjoy a strong standing in the community, should not.)

In CS, journals tend to operate a bit closer to this model than conferences, which I think makes the journal submission and revision process more meaningful. To conclude, I don't think getting rid of rejects is the solution, but rather editorial policies that encourage reviewers to either expose fatal technical flaws in a paper or help the authors improve the presentation to achieve a better end-result.

> So I don't think accepting everything would be ideal, but I'm more than happy to try.

An important caveat is that this experience isn't accepting everything; it's publishing everything (or at least, everything that satisfies the desk check and is sent out for peer review). The whole point is separating evaluation from publication; they'll still distinguish works that they think are significant and rigorous from those they do not, but all works will be accessible. That's the mental shift they're trying to make happen.

(Disclosure: I volunteer for a project with a similar goal, that has received funding from eLife in the past: https://plaudit.pub.)

as a long-time NeurIPS reviewer: the problem is that most submissions are kind of "meh", and everyone knows it. They're usually well-done papers (in the sense that the math checks out and the empirical evaluation is is done well enough), but it's pretty obvious that the paper isn't going to be influential. Mostly, these are papers of the "this is a cute idea that'll give you 0.1-2% improvements on some benchmarks people usually look at" variety. So depending on how strongly a reviewer believes that this might actually become influential if someone REALLY, REALLY digs down on this idea or not, you'll give it a "barely passing" or a "barely rejecting" grade. That's what most of these 60% usually are.

I don't think the problem here is actually the "broken peer review system". I think this is a fairly natural development: if your PhD advisor insists that you have a few "good papers" before they'll agree to let you graduate, and "publication venue" is used as a proxy for "is a good paper" (which is very reasonable, because it might take years before better proxies such as "citation count" give a good signal), then you'll submit your paper to top tier venues and hope for the best. I don't really know how to fix this: Establishing things like TMLR (Transactions of Machine Learning Research -- a rolling-release journal meant to have roughly the quality of NeurIPS) might be a good way forward. Students get their stamp of approval by publishing there and NeurIPS can significantly increase their acceptance threshold. But once you do that, you'll risk that TMLR doesn't count as "good enough venue" anymore....

> (1) there is something very broken for about 60% of submissions

My conclusion is that there is an inherent subjectivity involved in judging paper "quality". Which shouldn't be that surprising; if there were objective criteria we probably wouldn't need humans to do any judging. I dunno if you want to call that a broken system or not, but if this outcome is unavoidable then trying to change it won't improve anything. (I suspect that 60% isn't some magic lower limit and improvement is possible, but I suspect the lower limit is 20-40% at minimum).

Realistically, this means that a single acceptance/rejection shouldn't decide the fate of a persons whole career. And they generally don't, it will be an average of many.