Easter eggs from builders when deploying a LAN or why technical supervision is needed
When you install a network at a newly built facility, in an ideal world, the builders have already prepared the premises for you, installed the SCS, supplied power and done everything that is needed. In reality, synchronization between network integrators and builders is not always given due importance. What could go wrong?
I’ll tell you about a very illustrative case of such a desynchronization. I am Sergey Shulgin, an expert in network technologies at the IT company K2Tech. And with this story, I kind of hint at why technical supervision is needed in large projects and why wise customers write it into the contract with integrators at their expense.
In this story, everything will end well, because we are good at putting out fires and improvising in the moment. And then we draw conclusions. Stories like this teach us to build processes differently and see pitfalls in advance. I hope this story will help you avoid some organizational mistakes without having to learn the hard way.
How it all began
At the beginning of 2023, several companies within one large industrial holding, about 1000 people in total, decided to move from three offices into one. They turned to us to organize a local area network and Wi-Fi in their new office.
It was planned to place employees of different companies of the holding mixed together, on the same switchboards. At the same time, the companies' IT services worked differently and used different network configurations. The LAN should have supported this possibility.
Participants in this story:
We are K2Tech, an integrator. We planned the solution architecture and then carried out the implementation. On our part, 5 engineers and a technical manager participated in this.
Construction company that renovated the premises. In addition to beautiful walls, carpet, windows and doors, they were responsible for preparing the premises for the cross-connection, laying out the LAN, supplying power to the premises, equipment, and so on.
The customer is a holding company that is moving into a new office.
Plan-trap
At first, the holding approached us not for the network deployment itself, but for its planning. We completed this work in a fairly relaxed manner, within a month.
We audited existing equipment for reuse. They basically had Cisco equipment and we figured out how to reuse some of it.
For the rest of the equipment, we described the requirements so that the customer could independently select and purchase it. Plus, based on the customer’s criteria, we selected and recommended a specific vendor available in Russia, with whom we have already worked – Vector Technology. A nuance – we needed the vendor to reserve equipment worth several tens of millions of rubles for the project before the transaction, that is, for several months. We have a trusting relationship with both the vendors and the customer, so we were able to reach such an agreement.
The customer chose the proposed vendor. Taking this into account:
We decided to build the LAN on VA2100-48P access switches (22 pcs.), VC6100-24X core switches (2 pcs.) and reuse four old Cisco 9300 switches.
The wireless network included 56 VAP300-2X2i access points, 2 VNC-2000 controllers in clustering mode.
About our experience with the vendor “Vector Technologies” We have already dealt with their equipment on other projects and it passed tests well in our test laboratory. So it was a good workhorse for us. Functionally, the equipment performs well in terms of both wireless networks and LAN switches. We like the proactive position of the vendor: engineers connect remotely and, if necessary, go to sites when something needs to be tested, reproduced or configured. If necessary, modify the product. It is especially important for our clients that the vendor has accessible Russian-language support. The team is physically located in Moscow, answering email and phone calls, and traveling to sites. In Moscow there is a spare parts store with a good supply of equipment. One of the unobvious limitations of vendor equipment is that if there are more than 4 switches in a stack and at the same time you have blackouts, rebooting the stack begins to drag on. This can be cured by good UPSs or an architecture in which large stacks are simply not needed. |
Finally, we described in the form of technical specifications all the requirements for builders to provide space for the equipment, power supply, described the distribution of fiber optics, and so on.
We developed all this documentation in a month, handed it over to the customer and went our separate ways. Since the end of April, intensive site preparation and repair work have been underway. We were not involved in this, we were just happy for those who would connect the equipment – the guys will come with everything ready, right?
In May, we were invited to connect the equipment. Construction was already in full swing. We had a month to prepare for the residents to move in. To meet this deadline, the deployment of the LAN had to be carried out in parallel with the completion of construction work. Oh well, we've already done this. The equipment purchased based on our tip began to be delivered and stored in the most renovated and dust-free room – the kitchen.
Finding ourselves in the thick of construction, we realized that something had gone wrong.
How the builders did not receive our technical specifications and what came of it
Some interesting bureaucracy. The customer for our work was the IT department, and the customer for the construction work was the household department. In order for the technical specifications to reach the builders, it was only necessary for the IT and housekeeping department employees to be the best friends in the world and lend each other a shoulder in any difficult situation. In reality, everything was somewhat different, and the housekeeping department wrote its own version of the technical specifications to the builders, which was in no way connected with ours. So there was no need to worry about the carpet and chairs.
Now I’ll tell you why you should have worried.
Power supply. This is perhaps the biggest pain. According to the housekeeping department, the technical specifications did not include the necessary capacity for the equipment; they simply were not supplied to the premises. To solve this problem, we convened a two-day consultation with all key participants and the chief engineer of the building management company. We found a solution, but of course it didn’t look like what we had planned. We had to redesign all the racks in 2 weeks, this time in close sync with the builders.
Against this background, there were a lot of smaller jambs, like a twisted pair cable laid in a rack in such a way that it was impossible to mount anything on the 4 lower units (I wanted to put a UPS there).
Crossing. According to the plan, the cables from the workplaces were to converge into 5 cross-connect rooms. According to the plan, we just needed to run cross-connect switches and connect ports to them with short cables. But the builders decided to take advantage of a loophole in the wording in the technical specifications and save effort. Part of the LAN wiring was inherited from the premises by its previous inhabitants. The builders thought, why not give them to us? As a result, for some of the cables we did not understand which users were on which ports. I had to call them.
Well, in terms of premises, the LAN simply did not fail. So this temporarily happened in the office:
The builders handed over the documentation for the actual LAN to us the day before the employees moved into the new office. So we did most of the preparatory work by touch, without documentation.
Wi-Fi. The beauty of wireless networks is that builders cannot lay them crookedly – there are no wires. But the housekeeping department did not include brackets for access points in the technical specifications. On one of the floors, by luck, the brackets remained from the previous inhabitants of the room. On other floors, we did the fastenings ourselves: on time, cheaply, without SMS or registration. What an IT integrator has to do!
sticks and vines improvised building materials
Well, for some access points, the builders simply forgot to install cables, contrary to the documentation. So we couldn’t hang them up and figured out how to maintain coverage in other ways, for example, by tuning the signal strength of other points.
There were also a lot of small annoying moments, including broken keys in the rack cylinder. But to sum it all up, we were invited to deliver equipment in conditions in which this was impossible due to errors at the construction level. In order to urgently adjust reality to our technical specifications, they tried to give us supervisory functions. Only it was already too late. Our powers were not specified or regulated. Big problems had to be solved through general meetings of everyone. Smaller problems were solved thanks to our ingenuity, unbending optimism and willingness to do our own work, because without us it simply would not take off.
Whether long or short, by the deadline we deployed a LAN and wireless network, for which we achieved excellent signal coverage.
Check-in and friendly jokes
The arrival of the employees lasted 1.5 weeks. In terms of data and applications, the customer’s infrastructure was in the cloud, so migration only meant moving the jobs themselves.
In the evening, employees in the old office packed things, the customer delivered them at night, and in the morning the employees already came to the new office. Basically, seating charts were not established in advance, so we connected most of the employees in a live stream mode.
After the move, we corrected minor glitches that arose. Our favorite story: at some wonderful moment, a loop appeared in the network. We have found the network segment where this is happening. The guys from the customer’s team got there on their own feet. And this is what they found:
For what? We still don't know. I would like to believe that someone just decided to play a kind, friendly joke on us.
Technical supervision specified in the contract
In the end, everything worked out, and the customer came to us with a new project – a similar new building in which to deploy a network. In terms of equipment, we decided to build the network again on Vector Technologies.
In terms of everything that went wrong last time, we, together with the customer, arranged a debriefing. The main lesson is that uncoordinated work between contractors creates serious risks and can limit the capabilities of the IT infrastructure.
At the same time, creating emergency situations and the associated risks is generally a typical story for large customers. They set a time frame in which construction and infrastructure work must be carried out in parallel (sometimes also in parallel with the execution of contracts). You can be angry about it, condemn it, curse it – but it’s unlikely that this will ever change. I approach this philosophically: if the outrage cannot be stopped, it must be led.
So this time we will act as the technical customer for construction work in the part that ensures the commissioning of equipment. We translate the customer's requirements for IT infrastructure into requirements for premises, cabling, power supply, and so on, and these requirements are transferred to other contractors. Next, we carry out technical supervision – control and close contact with other contractors, their synchronization. The function of the technical customer and our powers for technical supervision are now spelled out in the agreement between us and the customer.
Well, in terms of time, now we have not 2 weeks, but several months to synchronize.
Invaluable life experience
The result of a particular implementation is, in a sense, boring: the project was completed successfully, on time, the customer was satisfied. But now you know how much work had to be done under the hood to make this happen.
From the point of view of life experience, we, together with the customer, took part in an instructive immersive show about out of sync between contractors. The customer learned why contractors should not be left in mutual isolation, and gained an understanding of the value of technical supervision (especially when he runs the risk of not being able to cope with this function given the tight deadlines of the project). Yes, the cause of the problems was the lack of communication between the IT and business departments of the customer. But the technical supervision function allows us to detect such difficulties in advance and search for a solution together.
Tell us in the comments if you had similar problems synchronizing with other contractors. And if you want to read more about the deployment of networks at a construction site and about emergency vendor replacement, then look, how we replaced Huawei with Maipu on the fly.
