Furious AI researcher creates a list of non-reproducible machine learning papers

On 14 February, a researcher who was disappointed to reproduce the results of a machine learning research paper opened a Reddit account under a username contribution. R / Machine Learning Subredit: “I just spent a week implementing a paper as a baseline and failed to reproduce the results. I realized today that after doing a little googly, some other people were not able to give results either. Is there a list of such letters? This will save a lot of time and effort of the people. “

The post struck a nerve with other users at r / MachineLearning, the largest Reddit community for machine learning.

“To compile a list of easy copywriters …”, a user replied.

“Probably 50% -75% of all paper is inaccessible. It is sad, but it is true, ”another user wrote. “Think about it, most papers are ‘adapted’ to bring to a conference. More often writers don’t realize that the conference they’re trying to go to isn’t great! So they worry about reproduction There is no need to do that because no one will try to reproduce them. “

Some other users had posted links to machine learning papers that they had failed to implement and voiced their frustration with the implementation of code, which was not required in ML conferences.

The next day, ContributionSecure14 created “Papers without code, “A website that aims to create a centralized list of machine learning papers that are not implementable.

“I’m not sure if this is the best or worst idea, but I felt it would be useful to collect a list of papers that people have tried to reproduce and thwart,” ContributionSecure14 written on r / MachineLearning. “This will allow authors to issue their own code, provide pointers or receive the paper. My hope is that it encourages a healthy ML research culture, which does not illuminate non-obstructive works. “

Reproducing the results of machine learning papers

machine learning Researchers regularly publish papers on online platforms such as AXXV and OpenerView. These papers describe concepts and techniques that highlight new challenges in machine learning systems or introduce new ways to solve known problems. Many of these papers find their way into mainstream artificial intelligence conventions such as Neurops, ICML, ICLR and CVPR.

Having source code to go along with a research paper helps a lot in verifying the validity of machine learning techniques and building on top of that. But it is not a requirement for machine learning conferences. As a result, many students and researchers reading these papers struggle with reproducing their results.

“Unreproducible work wastes the time and effort of well-meaning researchers, and authors must strive to ensure at least one public implementation of their work,” contributor 14, who prefers to remain anonymous, reported TechTalks In written comments. “Publishing a paper with empirical results in the public domain is futile if others cannot build from the paper or use it as a baseline.”

But ContributionSecure14 also acknowledges that machine learning researchers sometimes have valid reasons for not releasing their code. For example, some authors may train their models on internal infrastructure or use large internal datasets for pretraining. In such cases, researchers are not free to publish code or data with their paper due to company policy.

“If authors publish a paper without code due to such circumstances, I personally believe they have the academic responsibility to work closely with other researchers trying to reproduce their paper,” Says ContributionSecure14. “It makes no sense to publish the paper in the public domain if others cannot close it. There must be at least one publicly available reference implementation for others to construct or use as a baseline. “

In some cases, even though authors release both the source code and data of their code, other machine learning researchers still struggle to reproduce the results. This can happen for various reasons. For example, authors can obtain the best results from many experiments and present them as cutting-edge achievements. In other cases, researchers may have used tricks such as tuning the parameters of their machine learning models on a test data set to boost results. In such cases, even if the results are reproducible, they are not relevant, as the machine learning model is overfitted under specific circumstances and will not perform well on previously undiscovered data.

“I think it is necessary to have a reproducible code as a prerequisite to independently verify the validity of the results claimed in the paper, but [code alone is] Not enough.

Efforts to breed machine learning

Papers with code