Chapter XXVIII, Major Accidents
At noon the next day, Cao Song and Qian Dan stood downstairs at "ST Telecom" and were about to call the brothers to go to dinner together.
Cao Song dialed his mobile phone, and the more he dialed, the more nervous he became: "Brother Dan, I can't get through to the phone of a few people upstairs." ”
Qian Dan took out his mobile phone: "I'll call Lin Han." ”
On that day, Lin Han was not in "ST Telecom", but in the office of "Weizhong".
Qian Dan was also nervous: "I can't get through." ”
Cao Song said: "Brother Dan, I'll call you." ”
After that, he dialed Qian Dan's phone.
The two of them stood facing each other, and although Cao Song didn't use the speakerphone, Qian Dan could clearly hear the busy sound on his mobile phone.
There was no wind, time froze, and the two of them were sweating on their backs, turning around and running upstairs.
When they rushed into the computer room, they saw Ganfas and two customers surrounding Xiaowei, who was staring intently at the maintenance terminal. Cao Song shouted: "Xiaowei, what's the matter?" ”
Xiaowei's voice trembled: "I don't know, it seems that the business is all broken, I am tracking messages and reading logs." ”
"Why didn't you inform us?"
"I don't know where you are, I can't get through to your phone."
Cao Song glanced at the maintenance terminal and immediately decided: "Don't look for the reason, you restart a few business processes, and I'll replace a few boards of the switch." ”
The two of them operated separately, one restarted the software process on the host, and the other switched the primary/standby board on the switch, and then, several people waited anxiously, and tens of seconds were particularly long.
Xiaowei shouted: "Hey, it should be good, I'll test it." ”
Before the words fell, Lao Zhong rushed in with a dark face, and Chen Yongsheng followed closely behind.
The old bell said loudly, "What's the matter? I'm meeting with the 'CEO' and saying that the business is all down and it's a problem with our prepaid system?! ”
Cao Song hurriedly said: "Okay, okay, we are dialing and testing, and it has been recovered." ”
Lao Zhong asked continuously: "How long has it been broken?" How many users are broken? What's with? ”
Cao Song replied: "Look at the log for thirty minutes, I don't know the reason." ”
Lao Zhong was very dissatisfied: "I don't know the reason? You don't know how to break it? Well, I don't know how to do it? ”
Qian Dan explained: "Mr. Zhong, we first restored the business by restarting and resetting, and then we will find the reason. When we deal with major incidents, we should prioritize resuming business and then finding the cause. ”
Chen Yongsheng, who was behind the old bell, spoke: "I don't know the cause of the accident? Will the accident happen again? ”
Qian Dan hurriedly said: "We are guarding, and we will all stay here until we find the reason." The accident has been reported to the family, and the experts of R&D have come up to analyze it. ”
Lao Zhong asked: "Which leader in the family reported the accident?" Did you call Mr. Wang? Immediately arrange the support of the best R&D experts, quickly find out the reason, I will go back to the 'CEO' first, and call me as soon as there is progress. ”
Chen Yongsheng followed Lao Zhong out and turned back, pulled Qian Dan aside, lowered his voice and said: "It's best not to be the cause of 'Weizhong', especially not the man-made accident caused by the scene, even if it is 'yes', you have to make it 'no', otherwise you will be 'dead'!" ”
A major accident in a communication network usually refers to a situation where the phone cannot be connected, the text message cannot be sent, and the network cannot be connected due to equipment failure. Such a major accident will not only cause economic losses and brand damage to telecom operators, but also may have a bad social impact.
A communication network cannot always be zero accidents, but it must be able to recover from accidents in time, and the same accidents must be avoided repeatedly.
"Weizhong" has a strict accident handling and backtracking mechanism, after each major accident is completed, we must first carry out technical backtracking, find all the causes of accidents, identify the root causes, and then manage backtracking on the basis of technical backtracking, find the improvement points in the business process and management mechanism, and hold the responsible person accountable accordingly.
The root cause classification of major accidents is either "non-supplier-caused accidents" caused by customers or third-party equipment on the network, or "supplier-caused accidents" caused by equipment reasons of "Weizhong" or operation errors of on-site service personnel.
For example, more than a year ago, after a major man-made accident in a Yemeni subsidiary, the engineer was dismissed, the service director of the subsidiary was removed and transferred, and even Lao Han and Lao Xie of the regional department were jointly and severally liable and were criticized and fined a "huge amount of money".
A few people no longer have lunch, and they have forgotten about dinner.
They were busy until late at night to sort out the ins and outs of the accident, the core data network of the prepaid system of "ST Telecom" was not effectively isolated from the maintenance operation network of the customer's maintenance personnel, and the network of the customer's maintenance personnel was not effectively isolated from the office network of the customer's office, and it was the abnormal network traffic on the office network, that is, the so-called network storm, that caused the core system to be affected, which led to the accident.
The accident was interrupted for 30 minutes, and 500,000 calls were affected, which means that 500,000 mobile phone users in Sudan were unable to make calls for 30 minutes at noon that day.
The trigger for the accident was a cyberstorm in the customer's office network, and the customer's maintenance team didn't want to take the account on their own. Together with Weizhong's team, they wrote a report that attributed the cause of the accident to the management of their office network.
In addition, Chen Yongsheng ran up and down to do the relationship with several key customers, and the management on both sides did not have much dissatisfaction with Cao Song, but only ordered the technical teams of both sides to complete the rectification of the data network.
Two days later, Cao Song and Qian Dan convened all members of the team to hold an accident summary meeting.
Xiaowei explained the accident retrospective report, he said: "Fortunately, this time the accident was caused by the customer, and the impact was not too great, we must be vigilant and not have another accident." ”
Cao Song stood up and walked to the door to look outside, closed the door, and couldn't wait to return to his seat before speaking: "There are no outsiders, don't be a customer because of the accident, my reputation was almost ruined in Sudan!" This is a man-made accident, a few years ago in China because the network was not isolated, the customer maintenance personnel on the maintenance terminal with 'Kingsoft Shadow Ba' to watch movies paralyzed the network. A few years ago, the company issued an early warning to require the whole network to rectify, what year is it now? It's a shame that it's in Sudan, and if this kind of accident happens in the country now, I'll die. It's all my fault, and if I don't take it as an example, I'll treat everyone to a good meal tomorrow. ”
Qian Dan said: "This thunder was stepped on by us and was not left to future generations." Not only the data network of 'ST Telecom' must be rectified, but we must organize and inspect all data networks in North Africa and the Middle East, and all hidden dangers must be rectified. In addition, I have pondered in the past two days, we need to summarize the elements of maintenance work, increase vigilance, and grasp the 'three K's', the first 'K' is the 'Key Network', we will identify the key networks that need to be protected in the region, and focus on monitoring; the second 'K' is 'Key Actions', which clarifies the key actions to be taken in the daily maintenance of each network, including daily inspections; and the third 'K' is 'Key' Time', each network must have an emergency plan, and the maintenance personnel must be familiar with the response methods in the event of an accident, and they must also practice from time to time. ”
Lin Han said: "'Three Ks', good! I'm going to mention a 'Key Action' now, all brothers must have two mobile phones in the future, and they must be mobile phones of two different operators, in case of another accident where the business is completely cut off, they can no longer contact each other. ”
Xiaowei said: "I also mention a ......"
Since Cao Song has expressed his attitude and wants to invite everyone to have a good meal, there is no reason why everyone should not enthusiastically support it.
They went to a newly opened Chinese restaurant in Khartoum. Cao Song took Qian Dan, Lin Han and others to arrive first, and after a while, Chen Yongsheng took Xiaowei and Xiaojun and walked into the private room with a box of beer alone.
Qian Dan looked directly at the three cases of beer: "Tsingtao beer, where did you get it?" ”
Chen Yongsheng said proudly: "Do you still miss your bottle of Chivas Regal?" I didn't swagger in my suitcase anyway. ”
Cao Song said on the side: "He bought it from the people of 'China Railway', and he has treasured it for a while, and he is willing to take it all out today?" ”
"Of course, my family is all here, and I can't help it if you can't drink yet."
After a few rounds of drinking, Qian Dan felt that his stomach was swollen and got up to go to the bathroom.
Walking to the bathroom door, he was about to push the door in, but the door was pulled open from the inside, and a person came out, almost not bumping into him.
He looked fixedly, but the person was Zhang Wang.
When Zhang Wang saw him, he also had a surprised expression: "It's such a coincidence, you also came to Sudan to deal with the accident?" ”
Qian Dan opened his mouth and asked, "Who told you that we had an accident?" ”
He subconsciously put the accent on the word "who".
Zhang Wang walked to the washbasin, lowered his head, and washed his hands slowly: "Who else will tell me?" You have a 30-minute break at noon, 500,000 people can't get through to the phone, and the Sudanese people don't go to the customer's business hall to smash your shitty luck. ”
"You know that well?" Qian Dan took advantage of the wine, and his tone was obviously unfriendly.
"I go to 'ST Telecom' and find someone, and you can see the accident analysis report you wrote, what do you think?" Zhang Wang turned off the faucet and shook his hand, still not looking at Qian Dan.
Qian Dan had something in his heart and didn't know how to say it.
He pushed open the door of the bathroom, stopped again, and suddenly turned around and said a clueless sentence: "Brother Wang, don't keep pestering Zeng Zijian!" ”
Zhang Wang finally turned around and looked at Qian Dan, still slowly: "Brother, have you drunk too much?" What do you mean I keep pestering Zeng Zijian? Do you think I took Zeng Zijian to the ditch? I can tell you responsibly that I am not qualified enough to show him the way. It seems that you don't know your good friend, but he is much better than me. ”
As soon as Qian Dan returned to the private room, Chen Yongsheng shouted drunkenly: "Old Dan, where did you go?" The brothers said you urinated, I don't believe it, but is Dan always like that? Besides, if you want to escape to a good home in Cairo, you may be able to escape to a good home, and there is nowhere to escape in Khartoum, right? ”
Qian Dan said: "I saw outside that the people of 'Company F' were also eating here, and everyone was quiet, because the wall had ears. ”
Chen Yongsheng said: "What's the sound? Let's just talk about work. ”
He took Qian Dan's arm, pulled him closer, and whispered: "Sudan has a decisive big project in the second half of the year, we are guiding 'ST Telecom' to rebuild the network and go on '3G', Lao Zhong originally wanted to operate quietly as a confidential project, but he couldn't hide it, and he was still known by 'Company F' and wanted to bid for it." Lao Zhong is determined to win and wants to drive out 'Company F' in one fell swoop! ”
"When will the results be available? Do I need to prepare for the delivery of resources in advance? ”
"It's not in a hurry, it's not that fast, but it's definitely going to be won before the end of the year. So it's been very sensitive lately, and you must not have any more accidents. Lao Zhong was very careful, didn't you say that he had four mobile phones that day? One is that the more he thought about it when he was running this project, the more uneasy he became, so he bought another mobile phone, and the people who knew the mobile phone number plus the leaders of the regional department did not exceed five people, so as to prevent people from eavesdropping on him after knowing the number. ”
Qian Dan said: "Eavesdropping? It's getting more and more exaggerated! ”
Having eaten and drunk, they went back to the office in their usual way.
Lin Han put on headphones and started talking on the phone, Qian Dan and Chen Yongsheng sat together and chatted, and Cao Song stretched his waist and walked to the backyard. Before long, his roar rang out in the backyard:
Grasp every minute of your life
Give it all to the dream in our hearts
Do not experience wind and rain
How to see a rainbow
No one succeeds casually
Old Zhong walked out of his small office and walked out the door, he was in a hurry, and when he heard the singing, he stopped and said loudly, "Damn! Crazy one? It's Cao Song, right? Chen Yongsheng, please have a meal, relax, don't go crazy. ”
The people in the office laughed, and Chen Yongsheng replied, "Just finished eating, Cao Song drank well today." ”
Lao Zhong said again: "You should be careful, the company only looks at the results and contributions." With a good result, everything we do is right, and with a bad result, everything we do is wrong. 'Weizhong' has always had only true heroes, and there are no tragic heroes. ”
With that, he disappeared outside the door.
Qian Dan was thinking: In the past two years, many colleagues have come overseas from China, some people come out after being fully prepared like themselves, and some people like Cao Song have accidentally changed from "short trips" to "rooted in the world", but everyone is here, then safe, silently struggling in every corner of the world, it is really not easy.
Chen Yongsheng was thinking: Who said that the Chinese traditionally only guard their yellow land and have no adventurous spirit? The monks of the Tang Dynasty traveled thousands of miles to the Western Regions, Zheng He's fleet traveled to Africa, and the 12-year-old children of the late Qing Dynasty formed groups to sail across the sea to study. Although for hundreds of years, the Chinese have walked too slowly and too hard.
Lin Han took off his headphones on the side and walked to Qian Dan's side mysteriously: "Boss, let's go outside and tell you something, I'm so shocked!" ”
Qian Dan followed him inexplicably all the way to the outside of the courtyard and stood on the side of the road at the gate of the courtyard.
Lin Han couldn't hold back the voice in his throat and eyes, "Damn, it's really infernal affairs!" The regional department caught an inner ghost. ”
Qian Dan felt a sudden chill at the tail vertebrae, rushing all the way down his spine to the top of his head. He could feel the tremor of his voice: "Who did you catch?" Who is the inner ghost? ”