I've been trying to imagine how the ML research and publication enterprise could be re-organized. Here are some initial thoughts. Feedback welcome! 1/

My proposal has four components. At the core would be a wiki containing three major components: (a) an annotated and organized bibliography of published papers, (b) an organized set of ML experiment designs and analysis pipelines (with code), 2/

(c) an organized set of mathematical results and analysis techniques for proving results in optimization and learning theory. 3/

This wiki would be maintained by the research community with the senior editors belonging to three panels: (a) subject expert panel of people who keep up-to-date within some research area, (b) methodology expert panel of experts in experiment design and statistical analysis, 4/

(c) mathematical panel of experts in the analysis of ML algorithms. Serving as a senior editor would be viewed as a prestigious career path with associated visibility and rewards. Senior editors would be well-placed to give keynotes and tutorials. 5/

Authors would be expected to consult this wiki to identify relevant research results and methods. Failure to cite relevant work or methods that appear in the wiki would be grounds for rejection. 6/

The second component: A submitted paper would include a structured abstract and a structured appendix. The abstract would state the motivation, research question, methods, and contributions. 7/

The appendix would contain (a) An explanation for which wiki categories (and hence, prior work) they think are relevant and which ones are not relevant (but might appear to be relevant to a naive reader) 8/

(b) Pseudo-code of the experimental design and analysis pipeline (with access to the underlying code). A filled out checklist confirming that they followed best practices (i.e., they did not commit any of the common mistakes). 9/

(c) Statement and proof of all formal results.

The purpose of this structure is the help the authors identify weaknesses prior to submitting the paper. 10/

Component 3: Every paper is assigned an editor/mentor whose job is to help the authors improve the paper. The editor reads the paper and provides feedback on the "story": Are the motivation and claims well-supported by evidence? Is the research question interesting? 11/

Will readers learn something? The editor checks that the appendix is complete (but does not check correctness of methods or proofs); the editor checks that the relevant prior research and experimental methods have been correctly identified and described 12/

If necessary, the editor can post queries to a quora-like system where the subject experts, methodology experts, and math experts can post answers. 13/

The editor sends their suggestions (mostly in the form of pointers into the wiki) to the authors. The editor may also determine that the paper is too flawed to be considered further (e.g., because the claims are known to be false or the authors are crackpots). 14/

After the authors have updated their paper in response to the editorial feedback, and the editor has approved it, the paper is released at this point onto a platform, such as @openreviewnet , that invites the research community to provide reviews and replications 15/

Component 4: Papers can be nominated for a deep correctness check if they are regarded as important or surprising by the research community. An organization is funded (e.g., by national funding agencies) to pay reviewers to conduct these checks. 16/

The paper is sent to three kinds of reviewers. (a) People working on similar research questions: These reviewers evaluate whether the research question, claims, and evidence are accurately described. 17/

(b) Methodology experts (if the paper performs experiments): these reviewers assess the correctness of the methodology and analysis including auditing the experiments via selective replication 18/

(c) Analytical experts (if the paper makes formal claims): these experts check the theoretical claims and proofs. The results of the check are published along with suggestions for follow-on research to address any problems that were uncovered. 19/

The overall goal is an "edited @arxiv_org" with a second level of scrutiny for important papers. This should scale much better than the current system, which pretends but fails to carefully scrutinize everything /20

Equally important is that the community creates and maintains a wiki that captures the current state of knowledge in the field and gives authors better tools for "doing research right". Maybe this will solve the problem of the research explosion in ML? end/