"Mom, I won't go back this year, if it goes well, I'll go back to see you early next year."

"I'm not playing outside, and I'm not dropping out of school, don't be nervous, I know what I'm doing."

"You don't have to pay me, and I'm not starting a business, but I need to focus on solving some problems during this time. At the end of this year, you can rest assured, at the end of this year. If it doesn't work, I'll listen to you whatever you say. ”

"Okay, okay, remember to tell my dad to go out and exercise more, you pay attention to your body!"

"Back, back. Be sure to go back during the Chinese New Year. ”

After hanging up the phone, Meng Fanqi leaned against the wall and let out a long sigh.

I really couldn't explain these things to my parents, and if it weren't for the fear that Dean Fu would help, he wouldn't have planned to take a leave of absence.

Now that my parents know about the current situation, they must be worried. In the last few calls, no matter what I say, my parents have always been unable to put their minds at ease.

Meng Fanqi could only rely on the deadline at the end of the year to appease them, promising to listen to them for anything as long as they could not achieve outstanding results this year.

Looking at Don Juan, who was gloating in the room, Meng Fanqi didn't have a good breath and said, "What are you laughing at." ”

Don Juan instantly made a serious expression, but no matter how he looked at it, he had a feeling of three points of yin and yang, which made people very unhappy.

------------------------

Later in the day, Meng Fanqi received Alix's reply, and after looking at the time, Alix should have replied to the email as soon as he got up, which can be said to be quite diligent.

This is close to the start of this year's tournament, but Alix himself has no plans to continue to form a team to participate in this tournament.

It's just that although my brother is not in the rivers and lakes, the rivers and lakes are full of legends of my brother.

The 13-year Microsoft Research team, the National University of Singapore team, and the Decaf led by Jia Yangqing, UC Berkeley and other teams were all deeply influenced by Alix's ideas last year.

According to the method introduced by Alix, Meng Fanqi quickly completed the debugging, and successfully migrated the batch normalization, Adam optimizer and other operations from his experimental environment to the cuda-convnet framework, and made some adjustments and adaptations.

It didn't take long for it to be up and running on CIFAR-10.

Next is the core competitiveness of this competition, the implementation of the residual network on this framework.

"THIS YEAR'S IMAGENET CHALLENGE DOESN'T REALLY HAVE A LOT OF NEW IDEAS, AND EVERYONE IS BASICALLY BASED ON REPLICATING ALIX'S ALGORITHM."

Of course, Meng Fanqi is not ready to re-implement Alix's algorithm last year based on the framework of Alix, like other teams, and many of the early algorithms have great redundancy in the number of parameters and calculations.

"Google's GoogleNet and Oxford's VGGNet were supposed to be developed next year, but after I enter this year, I'm afraid these two important networks in the history of AI will be stillborn, right?"

Not to mention now, as of the end of '14, redundant designs are still in large numbers with Google and Oxford algorithms.

It wasn't until 15 years ago that the man won the championship ImageNet-2015 with the residual network ResNet, casting the godhead of AI models in the era of deep learning.

On the one hand, it will become a milestone in the era of deep learning, because its core idea is how to make the deep network more 'deep'.

On the other hand, its simplicity of implementation and structure discards a large number of designs that seem to be reasonable and valuable to humans, but are in fact of little use.

The simple and easy-to-use structure was repeated repeatedly, which also greatly reduced the amount of development work required by Meng Fanqi.

AlexNet is an eight-layer structure, each with some independent design, and at this point, the 8th layer is already a revolutionary deep network.

Deeper networks are particularly difficult to train, so it wasn't until '14 that Google and Oxford pushed that depth to layer 22 and layer 19, respectively.

The idea of ResNet completely and fundamentally solves the problem that the network cannot be trained smoothly when the network is deepened. It makes it possible to train 50, 100, 150, or even 1000 layer networks.

"From 8 last year, to 100+ layers this year, even 1000 layers can be trained. From the perspective of 2013, this dose of medicine should be strong enough. ”

Not only is it a groundbreaking breakthrough in depth, but the performance of ResNet is just as impressive. It is the first structure with a Top-5 error rate lower than human ability, reaching around 4.6% for a single model.

If you combine several ResNets trained in different ways and average their outputs and then predict them, the TOP-5 error rate is even as low as 3.7%.

"IT'S ACTUALLY A STRANGE THING TO BE TOO ACCURATE ON IMAGENET." Considering that in later studies, a large proportion of the labels in this dataset are wrong, the closer to 100% accuracy, the more absurd it feels.

The Res of ResNet refers to residual, which is a residual, and to put it more clearly, it is a short circuit or a jump link.

To put it another way, assuming that the original operation is f(x), ResNet changes the calculation process from f(x) to f(x)+x.

This was also one of the earliest discussions between Meng Fanqi and Dean Fu.

This kind of branch link without any parameters, directly adding the content before the operation to the content after the operation, is the secret of the deep network's sudden ability to converge at a depth of thousands of layers.

Almost all AI articles after this have been difficult to bypass this idea. Kaiming himself has received 150,000 or 60,000 article citations in six or seven years just for this article.

It has to be said that this is an astronomical amount. You must know that even if you are a frequent visitor to the three world-class top journals of Nature Cell Science, the famous Chinese and American double academicians of science draw one, and the total number of citations of all articles so far is more than 50,000.

Of course, it would be biased to evaluate the quality of an article and the quality of the author based solely on the number of citations a paper has.

Many basic subjects are high and low, and there are few people who read and write. Naturally, the quotation will not go up.

However, as far as AI is concerned, which is a discipline with strong practicality and experimentation, if you compare it internally, the amount of gold cited is still quite high.

People will only vote on the code, and valuable things will continue to be iterated on. Techniques that are conducive to improving the performance of your own model on tasks will continue to be cited.

There has always been a saying in academic circles that the two sides of the theoretical and experimental sides look down on each other. It's not like this is out of nowhere.

After Kaiming established his historical position with several masterpieces, there are always a few people who like to say that "the idea of ResNet has been mentioned a long time ago", "what was decades ago is not a new thing", "this article is just a lot of traffic, and it is the same as a traffic star".

The sour smell is going to overflow the screen.

It's not that Meng Fanqi hasn't read some theoretical articles, but he personally thinks that in the new era of AI, anyone can say P words, and writing code is the ability. You say something that existed decades ago, why don't you write a ResNet that breaks through the human level?

Is the core code of ResNet difficult to write? It's not hard.

Based on Alix's framework, it only took him a few hours to reproduce basically all the structures in the ResNet paper.

Weakness and ignorance are not obstacles to survival, arrogance is.