[linux-elitists] How to Move a .COM in Two Months and Survive
Thu May 25 13:06:47 PDT 2000
[This write-up of last week's BayLISA meeting -- not to mention the
actual videotape -- might be particularly useful to any firms that
are planning to move to larger quarters, soon.]
How to Move a .COM in Two Months and Survive
BayLISA presentation 2000-05-18
Givan by Hal Pomerantz (Autodaq's recent move)
(notes by Rick Moen, email@example.com)
(Hal was hired to supervise the information-technology aspects of Autodaq's
move to a new building. He's a well-known network consultant, and a very
funny speaker. This set of hasty notes is mostly a transcription of Hal's
presentation slides, and cannot possibly do his presentation justice.
BayLISA will have a videotape of Hal's talk available
for rental, and it'll be worth whatever they're charging.
Telco: Order Your Circuits, 30 Days Minimum for T1
Don't forget to include some POTS lines:
Postage meter, elevator emergency 'phones, security system, postage meter.
Place orders as new installs rather than change orders! They never get
the latter right. Therefore, separate the hell of ending service from
the hell of starting it.
What about Your 'Phone Numbers?
Default assumption is new numbers. There is an extra monthly charge to
forward old numbers. 800 numbers can be retargeted at will.
New numbers are "easier", but you have to reprint letterhead, business
cards, etc. (but you'd need to do that for the new address, anyway).
If you do get new numbers, order way more than you think you'll ever need.
Order DIDs in blocks of 20. $8/block/month is going rate.
What Happens to Your Order?
CLEC (Competitive Local Exchange Carrier, e.g., ICG) Sales rep hands off
order to order entry dept.
Order entry converts paper form into electronic order for Provisioning.
Provisioning takes electronic order and generates ILEC/CLEC work tickets.
After 48 hours, ILEC (Independent Local Exchange Carrier, e.g., PacBell)
comes back with commit date 3-4 weeks out.
ILEC eventually installs local loop.
ILEC eventually cross-connects circuit properly in the central office.
CLEC performs end-to-end test, repeat until all bugs are fixed.
Multiple points of foul-up. Minimum 1 week, probably 2.
You get to test with your equipment, repeat until all bugs are fixed.
Total six/seven weeks!
Second Priority: Order 'Phones:
Want a PBX? Then, prepare to wait three months -- minimum ten-week
Key systems are cheaper and faster to get. Less functional; less cool
programming stuff. Expect to pay $1200/user for PBX, $800/user
for key systems without ACD.
Biggest discounts are _before install_, so front-load those orders!
Telephone systems are a razor-blade and razor deal: It's cheap until
you're locked in, and then they gouge you.
Be aware that reseller expects front-loaded payment schedule.
The way to survive is to front-load your purchase as much as possible.
Pick the Move Team:
You need at least: one facilities person, one finance person, and
one IT person.
You'll be interfacing with: real-estate broker, architect, contruction,
HVAC, telco, 'phone vendor....
You'll need to meet _weekly_, produce complete notes and action items for
Moving is a full-time job for everybody.
Finally, Pick Your Move Date:
Can't move in without 'phones or a network, so those items set your date.
Don't leave the move until to last possible moment either.
Make sure everybody's vacation schedule interfaces with move-in date.
Philosophy and Money:
For IT folks, mvoes present a chance to fix infrastructure problems.
Tremendous learning experience.
Great team-building scenario, providing a huge sense of accomplishment.
Opportunity to spend lots of money.
In this case, they replaced numerous NT-based systems with Unix.
Moving: the Bad News
Opportunity to spend lots of money.
Nothing ever goes as planned, everything takes longer than it should
and costs more.
No margin for error, stress levels high.
Survival Philosophies #1: Blame:
It's your fault, even when it isn't. Nobody will remember, as long
as you help make the problem go away.
If it's not your fault, it's nobody's fault.
"We're all on the same team": This attitude helps even with telcos.
Blame is incredibly counter-productive.
Survival Philosophies #2: On the Job:
"Trades" are like sysadmins: Nobody gives them credit or recognition.
They know more about what's going on than anybody else.
Treat them well -- you get better workmanship and they'll save your butt.
Food always helps. Listen to them, hang out with them, laugh at their jokes.
Big Ticket Items #1: The 'Phone System:
$750/user times 130 users is $100,000.
You'd think that includes the music on hold, but it doesn't.
Big Ticket Item #2: Wiring Plant:
4-plex CAT5 drops cost $200/station.
We did one drop per cube and two for each office and conference room.
Don't forget extra capacity near printers and copiers and for the front
Don't forget cross-connects from the MPOE to the primary demarc.
You'll also need racking and termination in your IT room and/or IDF.
Big Ticket Item #3: Backup Power:
How much battery time do you need?
Going with a fuel cell or diesel generator?
Labour can be 50% of capital cost (or more).
What percentage of your data centre will be on the UPS?
Don't forget this stuff takes up lots of room!
Put an analogue 'phone next to the dead-man switch. If you're using or
contemplating using one, you'll need it.
Big Ticket Item #4: Security and Fire Control:
Need to replace wet systems in data centre with appropriate fire control.
Put a 'phone next to the dead-man switch!
Electronic locks (expensive) for external doors, electric strikes for
You'll need to provide a Windows PC for badge programming, et alii.
Whatever you predict your budget to be for this project, double it.
Week of Jan. 10:
First tour of the building
Carve out the data centre (12'x 30')
Delegate data commorders
Quick immersion in telephony for Hal (who previously knew nothing about
the subject -- got quick course from veteran at the neighbouring firm,
Barry at Onebox. Barry says "You're a network guy, eh? What's the
problem with RIP? Hal says "Convergence." Barry says "You'll do.")
Week of Jan. 17:
Telco/data orders in (delivery late Feb)
Set move-in date for 3/03
First meeting with the wiring contractor
First construction meeting - building permit process longer than two months.
Establish HVAC loads for data centre and office space. HVAC contractor was
recommended by real estate firm, did fabulous job.
one ton of cooling = 12,000 BTUs per hour
Week of Jan. 24:
No permit (takes way too long, in San Mateo County), creating problems for
data centre: Had to circumvent permit process.
-- No raised floor in data centre.
-- Can't suspend anything over 4000 pounds from the ceiling.
- Can't remove sprinklers, either. Have to leave "wet" system in
Cet more bids, order phone system.
Floor plans delivered to wiring contractor, specs for copy areas, etc.
Construction error in data centre: Wall got built inwards, into the building,
making the data centre too narrow (2' passage). Therefore, had to knock
down drywall and rebuild.
Hal flies to Saudi Arabia for teaching.
Week of Jan. 31:
Hal's in Saudi Arabia
Getting bids on security system
Cubicle planning underway.
Otherwise, just ongoing construction.
Week of Feb. 7:
First cube plan published (8'x8' cubes)
Main stairway needs substantial redesign to meet fire codes.
Identified need to buy a refrigerator for kitchen.
Order UPS for data centre.
Place disconnect order on Centrex service at old premises.
Week of Feb. 7 Cont'd -- Initial Entropy:
HVAC contractor belatedly says the A/C will be "thin". "Thin" is bad,
in this context. Can't add more units without triggering permit process.
Decided to go with portable units until permit work could be done.
Problems getting data to all cubes:
Vertical cable runs to "floating" cubes.
Right angle turns for 40 cable bundles. What's that turn radius,
Network admin _quits_, and suddenly everyone realises that data comm
orders were screwed up.
PacBell hadn't gotten _any_ orders; MCI had gotten change rather
than new-service order. Was able to intervene in the latter at
the last minute, frantically calling until switch to change order
got approved; PacBell said it would be able to meet (revised) deadline.
Week of Feb. 14:
We need more seats -- second cube plan published (8'x6' cubes)
Marketing gets involved in lobby design -- oh joy!
Sign off on security system vendor.
Sign contract with moving companyi.
'Phone system design meeting.
Move date pushed back to 3/17. (Latest possible, as lease expires
Week of Feb. 14 -- More Entropy:
How big is that UPS?
UPS unit itself is reasonably small, but:
Battery bays turn out to consume 8 linear feet
Don't forget transformer and junction: too heavy to hang. (No permit.)
Centrex disconnected service on Feb. 17 instead of the planned Mar. 17 (oops!)
(PacBell was able to resurrect within one day.)
Week of Feb. 21 (move is now 4 weeks off):
Third cube plan published (slanted cube clusters) - need exact dimensions!
Eyeball-to-eyeball meeting with furniture, wiring, and electrical contractors.
(You need to have one of these, and make sure everyone's clear one what
everyone's doing, and in the right order.)
Still messing around with lobby design. Marketing department can't
make up its mind.
How long can it take to get a 'fridge? $4000, with _4 week_ lead time.
Need more capacity in 'phone system. More voice-mail, more ACDs.
Week of Feb. 21 - It Gets Worse:
MCI circuit disconnected because old "change order" not deleted.
Snafu about processing of the changes when the net admin quit.
Re-routed 800 # through the data circuit, to tempoariliy fix.
Uh, where are my PRIs? No copper, even. Can't reach the PacBell
sales guy, other requested circuits are in limbo.
For that matter, where's my new Internet service?
Week of Feb. 28 (move: 3 weeks):
Cube plan finalised
Marketing is still designing lobby. They want granite tiles. Lead time
for granite is 5 weeks. Salesman furnishes tiles on spec, regardless.
Carpet is scheduled to go in, but they're still drywalling.
It's the simple things that end eating up your time - music on hold:
Support wants to be able to record messages occasionally for the
hold music. Worse, the marketing people get involved: Want
1950s surf music: "Surf the Web" theme. Hal tries to find some. Records
20 minutes of music from a friend, Marketing people hate it. They
want Beach Boys, Surf City.
What _are_ those new 'phone numbers? Have to order business cards, etc.
PRI still hasn't come in.
Week of Feb. 28 -- Problems #1:
Need to relocate electric boxes up into the ceiling beams.
Miscommunication on data centre electrical -- what's a rack, anyway?
Contractors put _one_ 20A circuit on each of the four racks, instead
of four on each rack: They had assumed "rack" refers to a row of
racks. Fixed by the end of the week.
Still cannot reach the PacBell sales rep.
Week of Feb. 28 -- Problems #2:
MCI circuit disconnected twice:
MCI re-uses our pair for a different customer, since it's "free"
(reading the erroneous prior disconnect order).
PacBell disconnect at CO due to old ticket
And on the ICG front:
Order "lost" due to re-org. Fellow who was supposed to have set
it up had been fired, and had done nothing. Hadn't even been
provisioned, yet. Instant escalation to ICG regional manager.
New commit date of 3/3
PacBell aborts install due to "bad address": They had driven to the
wrong street number, gave up, cancelled the order. PacBell rep finally
calls back, says she had been in a bad auto accident, and has been in
a hospital bed, badly injured. Is obviously zonked out on pain
medicine. This explains the lack of call-back. Nobody at PacBell
covered for her.
Week of Mar. 6 (move - 2 weeks):
Create "punch-list" of items needing attention.
Building inspector visit.
Order Cat5 patch cables. Telco, data, and data crossover must be
distinct colours. Yet another colour for cubicle patch cables.
Equipment installs begin.
Hal needs a building key.
A digression on Patch Cables:
Rule #1: You always need a foot more than you expect on either end.
Corollary to Rule #1: Patch cables that are 2' or less are useless.
Result: Would anyone like to buy a bunch of 2' cables in various colours?
Week of Mar. 6 - Problems:
Hal finally looks _up_ - must move some piping and conduit in data centre:
Electrical people put conduit for UPS exectly where duct for AC would
have to go.
HVAC water drip ended up terminating inside the data centre. Cured
by redirecting it out through a hole drilled in the wall, and onto the carpet.
Delivery time on patch cables: Graybar messes up, missed by a week.
Long ones were back-ordered.
Larry, Moe, and Curley wire the doors. Guys are saying "I wonder
which door these wires go to?" Obviously don't know what they're
What happened to the locksmith?
Locksmith finally shows up, on Thurday is 3/4 done, says he
cannot continue, has to go to a trade show the next day. Hal calls
the nice lady at SecurityLink, she says "Well, that's weird."
She finds out that EVERYBODY has gone to the show in Las Vegas.
Week of Mar. 6 - the Saga Continues:
On the ICG Front:
3/30 delivery date from PacBell
ICG won't dispatch until circuit is delivered
Rush hour traffic prevents full testing -- tech sent at 4:30 from
San Francisco, arrives in Menlo Park at 5:30pm, gets off work at 6.
Initial testing is not looking good -- no carrier.
Frame relay install finally rolling in PacBell. What are my IP addreses,
please? The department Hal's dealing with doesn't know.
Week of Mar. 13 (Move Week):
Final HVAC testing
Janitors start cleaning
Security system straightened out - SecurityLink sends its best guy
to rip the panel apart and re-wire everything.
Phone system install. Best telco guy available is on-site, but bad
news is he's always getting called to consult on other jobs.
Hal moves into local hotel, since he's on-site practically all the time.
Week of Mar. 13 -- Telecomm Hell:
MCI circuit up but can't bring down current connection to test
Frame relay come in Thursday night
ICG vs. PacBell
More testing ("Irvine, we have a problem") Irvine is where the top
ICG people are. PacBell hasn't done the cross-connect in the CO,
says Hal, and he turns out to be correct.
"Hey, the problem's in the CO!"
Are we going to have 'phones? Autodaq CEO is hanging over Hal's shoulder,
Full backups of everything! Did you do _test restores_?
Place your troops! Make sure everyone knows where he's going to be.
The dangers of fixed-price bids: The movers didn't plastic-wrap the dollies.
Dollies get lifted up to 2nd floor on a jack; PC immediately tumbles
to concrete below, shatters. Office manager sees this happening,
and says "I'll bet that's my PC." She's correct. Case and
motherboard shattered into small pieces. This PC's hard drive
has the cheque-writing software, and isn't backed up: She's the one
who has to write the cheque for the movers. (Karma's a bitch,
sometimes.) The hard drive, an IBM, turned out to be undamaged.
Can't have too many maps. Had one on the wall in the data centre,
while doing the cross-connects for the workstations.
How many ethernet ports do we need? Engineers had been using hubs in
their old cubicles. Need was therefore underestimated.
Everything always takes longer. Thu/Fri/Sat/Sun Hal billed 50 hours.
Half a dozen people worked full-time all weekend.
"It doesn't take that much longer to do it right!" Spent another 45
minutes, bone-tired, getting all the covers back on the Panduit
units. Worth it when inspection time came on Monday morning.
ICG: The End of the Story:
When the amber lights on the phone system go out, that will indicate
that the PRIs are up.
Amber lights go out Saturday afternoon.
But wait: There's more:
Long distance services _not programmed_. Had to use long prefixes.
Translation problems on local numbers. Couldn't dial some areas.
Incompatible PRI orders. Phone system could not deal with 23B+D
lines. Second PRI was therefore effectively not working. Spent weekend
trying to understand and document the problem.
Service fully working Sunday night.
Founders of the company are walking around the building all weekend,
not bothering Hal but watching closely. CEO comes by during lunch
on Sunday, and asks exactly one question: "Are we going to have 'phones
on Monday?" Hal says "Yes, Adam, we will." CEO leaves, gets in his
car, and goes home, as that was all he needed to know.
Recording the "Voice of Autodaq" for 'phones -- got to get the trades to stop
hammering for a while so the woman in question could record.
Need card access on upstairs door! Neglected to realise that there
was another egress there. No building security until that happens.
Need third row of racks.
Need more ethernet switches.
Hal flies to Orlando for SANS 2000.
Disconnect service and retrieve equipment from old building.
Telco refused to process this in less than 30 days. (Decision to
go with new-service order rather than change order gets validated
Class Acts and Thanks:
Autodaq: Suzanne Taafe the office manager, Peter Kelly the CFO,
Graham Breeze the IT manager
Matsart Consulting: David Howard
Onebox the neighbour: Barry Bouiler, who gave Hal the telecom education
ICG - Ken Bown Targee Oswald
MCI - Art Gerans
PacBell - Aaron Coles
Teledynamic -- Tony Szalo (the tech you want), sales rep whose name I missed
Blatch Construction - Robert "Skelly" Skelenger (quick, good work), Bob
DES Architects - Elke MacGregor (good sense of humour)
CableTech -- Bob Correly, Patrick Sumner: can't recommend these guys enough
Security Link: Dlo Spear -- did a slow burn when she found out how
badly her staff had burned Hal.
DI Electric - Mike ???
Deer Run Associates -- Laura (his wife), for putting up with him
Black Chatsworth racks!
Digi Etherlites (32 port model)
GNAC console server software
Raritan keyboard-video-mouse (KVM) switches: 16-port, connects
up to four virtual consoles, 16x16 machine connections.
Can remote anything over Cat5 wiring, over to the IT area without
having to go into the data centre. Extremely expensive, but
useful, especially in an NT environment.
Flexibility and a sense of humour are key survival traits.
"Every human encounter is an opportunity to be kind".
"We made it! We made it! We made it!"
Cheers, It is by caffeine alone I set my mind in motion.
Rick Moen It is by the beans of Java that thoughts acquire speed,
rick (at) The hands acquire shaking, the shaking becomes a warning,
linuxmafia.com It is by caffeine alone I set my mind in motion.
More information about the linux-elitists