On 14 February, a researcher who was disappointed to reproduce the results of a machine learning research paper opened a Reddit account under a username contribution. R / Machine Learning Subredit: “I just spent a week implementing a paper as a baseline and failed to reproduce the results. I realized today that after doing a little googly, some other people were not able to give results either. Is there a list of such letters? This will save a lot of time and effort of the people. “
The post struck a nerve with other users at r / MachineLearning, the largest Reddit community for machine learning.
“To compile a list of easy copywriters …”, a user replied.
“Probably 50% -75% of all paper is inaccessible. It is sad, but it is true, ”another user wrote. “Think about it, most papers are ‘adapted’ to bring to a conference. More often writers don’t realize that the conference they’re trying to go to isn’t great! So they worry about reproduction There is no need to do that because no one will try to reproduce them. “
Some other users had posted links to machine learning papers that they had failed to implement and voiced their frustration with the implementation of code, which was not required in ML conferences.
The next day, ContributionSecure14 created “Papers without code, “A website that aims to create a centralized list of machine learning papers that are not implementable.
“I’m not sure if this is the best or worst idea, but I felt it would be useful to collect a list of papers that people have tried to reproduce and thwart,” ContributionSecure14 written on r / MachineLearning. “This will allow authors to issue their own code, provide pointers or receive the paper. My hope is that it encourages a healthy ML research culture, which does not illuminate non-obstructive works. “
Reproducing the results of machine learning papers
machine learning Researchers regularly publish papers on online platforms such as AXXV and OpenerView. These papers describe concepts and techniques that highlight new challenges in machine learning systems or introduce new ways to solve known problems. Many of these papers find their way into mainstream artificial intelligence conventions such as Neurops, ICML, ICLR and CVPR.
Having source code to go along with a research paper helps a lot in verifying the validity of machine learning techniques and building on top of that. But it is not a requirement for machine learning conferences. As a result, many students and researchers reading these papers struggle with reproducing their results.
“Unreproducible work wastes the time and effort of well-meaning researchers, and authors must strive to ensure at least one public implementation of their work,” contributor 14, who prefers to remain anonymous, reported TechTalks In written comments. “Publishing a paper with empirical results in the public domain is futile if others cannot build from the paper or use it as a baseline.”
But ContributionSecure14 also acknowledges that machine learning researchers sometimes have valid reasons for not releasing their code. For example, some authors may train their models on internal infrastructure or use large internal datasets for pretraining. In such cases, researchers are not free to publish code or data with their paper due to company policy.
“If authors publish a paper without code due to such circumstances, I personally believe they have the academic responsibility to work closely with other researchers trying to reproduce their paper,” Says ContributionSecure14. “It makes no sense to publish the paper in the public domain if others cannot close it. There must be at least one publicly available reference implementation for others to construct or use as a baseline. “
In some cases, even though authors release both the source code and data of their code, other machine learning researchers still struggle to reproduce the results. This can happen for various reasons. For example, authors can obtain the best results from many experiments and present them as cutting-edge achievements. In other cases, researchers may have used tricks such as tuning the parameters of their machine learning models on a test data set to boost results. In such cases, even if the results are reproducible, they are not relevant, as the machine learning model is overfitted under specific circumstances and will not perform well on previously undiscovered data.
“I think it is necessary to have a reproducible code as a prerequisite to independently verify the validity of the results claimed in the paper, but [code alone is] Not enough.
Efforts to breed machine learning
The problem of replication is not limited to small machine learning research teams. Even large tech companies that spend millions of dollars on AI research every year often fail to validate the results of their papers. In October 2020, a group of 31 scientists wrote Joint article in NatureCriticizing the lack of transparency and reproducibility in a paper on the use of AI in medical imaging published by a group of AI researchers at Google. “[The] The absence of adequate documentation methods and computer code inherent in the study effectively undermines its scientific value. This lack limits the evidence needed to validate and clinically apply future technologies to others, ”the authors wrote. “Scientific progress depends on the ability of independent researchers to investigate the results of a research study, reproduce the main results of the study using its materials, and build upon them in future studies.”
Recent years have seen a growing focus on the reproductive crisis of AI. Notable work in this regard includes the efforts of machine learning scientist Joel Peanue at McGill University in Montreal and Facebook AI, to push for transparency and replication of machine learning research at conferences such as Neurops.
“Better reproducibility means that it is much easier to produce on paper. Often, the review process is small and limited, and the real impact of a paper is something that we see much later. Remains on paper, and as a community we have a chance to build on the work, check the code, have a critical eye for contribution, “Peanu Told Nature In an interview in 2019.
At Neurops, Pinneau has helped develop standards and procedures that can help researchers and reviewers evaluate the ability to reproduce machine learning papers. Their efforts have resulted in increased code and data submissions on neurips.
Another interesting project Papers with code (Where it gets its name from papers without paper), a website that provides implementation for scientific research papers published and presented at various locations. The coded papers currently host the implementation of over 40,000 machine learning research papers.
“Paperworthcode plays an important role in exposing papers that are reproducible. However, it does not solve the problem of unrestricted papers, ”said ContributionSecure14.
When the research paper learning machine does not include the implementation code, other researchers reading it should try to implement it themselves, it is a non-trivial process that can take several weeks and ultimately result in failure .
“If they fail to implement it successfully, they may reach out to authors (who may not respond) or simply give up,” said contributor Secure14. “This may be the case for many researchers who are not aware of prior or ongoing efforts to reproduce the paper, resulting in several weeks of productivity being collectively destroyed.”
Papers without code
Papers without code include a Submission page, Where researchers can present non-obstructive machine learning papers with details of their efforts, such as the amount of time they spent replicating the results. If a deposit is valid, the paper without paper will contact the original authors of the paper and request clarification or publication of the implementation details. If the authors do not respond in a timely fashion, the paper will be added to the list. Unproven machine learning papers.
“PapersWithoutCode solves the problem of centralizing information about prior or ongoing efforts to reproduce a paper and allows researchers (including the original author) to come together and implement public implementations , “Said ContributionSecure14. “Once the paper is successfully reproduced, it can be published on PapersWithCode or GitHub where other researchers can use it. In that sense, I would say that the goals of Paperswithoutcode are synergy with that or paperswithcode and the ML community at large. “
The paper is expected to help establish a culture that encourages fertility in machine learning research. So far, the website has received more than 10 requests and an author has already promised to upload its own code.
“I know this can be a controversial topic in academia and the top priority is to protect the authors’ reputation while serving the wider ML community,” said ContributionSankar14.
A paper without code can become a center for creating a dialogue between the original authors of machine learning papers and researchers who are trying to resume their work.
“Rather than being a static list of unproproducible work, the hope is to create an environment where researchers can collaborate to reproduce a paper,” said ContributionSecure14.
Reproducible machine learning research
For example, if you are working on a research building on work done in another paper, you should try the code or machine learning model yourself.
“Do not construct claims or ‘insights’ that could possibly be unfounded simply because the paper says so,” says ContributionSure14, including that it contains papers from large laboratories or work that have been called a prestigious conference. Has been accepted in.
Another good resource is Professor Pino’s “Machine learning reproducibility checklist“The checklist provides clear guidelines on how to create, code, and data a machine learning paper and is reproducible for other researchers.
ContributionSecure14 believes that machine learning researchers can play an important role in promoting a culture of reproduction.
“There is pressure to publish very deeply and on academic depth and ability to reproduce, there are not many checks and balances to stop this behavior,” said ContributionSecure14. “The only way this will change is if the current and future generations of legislators prioritize quality over quantity in their own research.”
This article was originally published by Ben dixon On TechTalksA publication, which examines technology trends, how they affect the way we live and do business, and the problems they solve. But we also discuss the bad side of technology, the deep implications of new technology, and the things we need. You can read the original article Here.
Published March 6, 2021 – 09:00 UTC