Your last generator Maintenance Run

[Industrial Toys]

I sure hope those people with the generators don't get cancer, messing with that toxic gasoline.

As far as cell sites go. There are so many here and they say they they can bring generators out to sites with failing batteries. Great for local outages but certainly not a strategy for widespread outages. High value, high traffic areas have many, many sites serving a small area. No way you could begin to have them all backed up or rotate generators. I'm guessing the point comes quickly when they just give up trying.

 

[Chilliwack Murray]

When there’s a large outage they just give up. Same is true for coax internet which require boosters throughout the system.

They try for a while and even put a portable up on the pole near me but they don’t have the resources to refuel it every 4-6 hours when the outage lasts for several days and/it affects 60 or 80k people. Then the cable, internet and cable company phone are all useless and you have to rely on over the air broadcasts.

 

[ultrarunner]

The church gets a nice monthly check and free WiFi for the cell towers which are almost invisible on the hillside.

During the second PSPS a large MQ genset was stationed to keep systems functioning... also surprised that competitors shared the generator as several providers use the church property... the 40kW was super quiet and little more than idle... the pastor said fuel is checked every two weeks

 

[3930dave]

There's the Thing.....

Done my share of ranting (possibly in this thread...).... about New Improved tech displacing older systems, at the price of reliability.

So I guess we're not supposed to have medical emergencies or serious accidents outside a low-traffic tower's limited battery capacity during grid-down ?

Yep, land lines can be interrupted, but in the 30+ years I've had mine, it's never been down. The COs were designed back in the olden days to have significant battery backup, and that same DC power ran the handset indefinitely......

Many folks (general public) don't care (or don't realize) what they've given up to get the baubles and trinkets in their hands today......

It is rather sad that the E911 performance of a phone system that was designed before I was born is still, today, much more likely to hit five9s than the cell I carry......

Eng Rant Off......

Rgds, D.

 

[Industrial Toys]

Do you know that they used to build the telco microwave towers and buildings to withstand nuclear blasts? There is an AT&T video somewhere called "It Shall Not Fail" All that has gone out the window in favour of building this increasingly complex, frail technology TRAP!

Both my hard wired lines failed about a week ago. I am the only one on the 13 pair cable that goes by my house, which used to be insuficient for customer demand. I didn't even bother to report the lines, which began working again within a day as mysteriously as they had failed.

 

[RoyKing]

Was talking to a tower service guy, and mention the backup power system. He said it probably won't work, the law made them install them, but doesn't make them test them. He said he doubts the batteries are even good.

 

[3930dave]

The level of backup depends on the importance of the traffic the site handles. Low priority sites with only cellular and data, particularly in rural areas with lower traffic will only have battery backup to last anywhere from 2 to 12 hours depending on traffic load.

Many sites have emergency communications share the tower and provide the financial support for a full backup system which is really a win win for everyone.

Thnx for all of that post CM...... puts some real #'s around what I suspected.

Rgds, D.

 

[3930dave]

Other than user prioritization, I knew there had to be more to cell services often going down fast in disasters..... just what we've covered here is concerning enough.

Comms should be part of any emerg planning..... at least have a battery powered broadcast radio around (my fav is a Grundig), and if you've never checked it out, getting an amateur radio license is a pretty easy task....

Rgds, D.

 

[3930dave]

Started reading on /. , about a Samsung fab getting shut down by a 1 minute outage.....

Samsung Chip Output at South Korea Plant Partly Halted Due To 1-Minute Power Glitch - Slashdot

Reuters article is pretty thin on details. One guy posted in that thread about some fun at work with their "automatic" system.... BTDT for guys here like cm and grs, but some good examples for the rest of us......

"(Quoted comment: I've seen UPSes choke during the flip to the diesel generator (in one case there was a battery fire). I've heard one place where I worked where the contractor who installed the generators messed up on the cooling system and the problem wasn't discovered until the generator was at full load for an extended period of time (massive grid power outage) I've heard of the generators just not starting when needed despite previous testing. You just never know.)

Where I work, we have a large UPS for our data center, to run the AC as well as the servers, router, and switches for the ~5 minutes or s it takes for the generator outside to fire up and stabilize. The generator automatically self-tests once a week too.

One morning the transformer on the pole at the road blew up. The UPS turned on, but the generator just started up and then almost immediately shut back off, and would not start up again. When we tried to manually start it, it started, and then just stopped again. Nobody knew what was wrong, our phone support was stumped. We had to scramble to shut things down orderly befrore the UPS ran out. Turns out the idiot that topped the fuel in the generator recently forgot to put the cap back on the tank, and the generator didn't like that. No one noticed the generator running too briefly during the self tests since then. (we had no software monitoring the tests) It was then that we discovered that we had no streamlined/scripted process for bringing down the critical servers, and forced our server guy into a mad dash to manually shut down things in order of criticality. (some of the less sensitive systems didn't get shut down in time) So that was a wakeup for us to streamline and automate that process for the next time we'd need it.

Then we had someone hit a power pole down the road, power went out, UPS came on, generator fired up and ran smooth, everything looked to be running fine. With the generator running fine outside, no one noticed the transfer switch didn't operate. 35 or so minutes later, the UPS exhausted and everything went down hard, including the SQL server and the VM box. (NOT good) The transfer switch was not part of the regular test, the automatic test was only firing up the generator, not making sure it took over once warmed up. And we had no alarms to notify us that the UPS was running longer than it should, we only got alarms when it was ~5 minutes from exhaust - not enough time to shut down much of the sensitive servers. Another wakeup.

We got a new UPS and transfer switch recently, and the generator is being monitored more closely now, but really the only way you know your power failsafes are working is to FULLY test them. (hard disconnect the power feeding the network center at the breaker panel and let it run an hour or so on emergency power) But you can't risk that during production, and you need to have staff on-hand in case you have to do something like an emergency shutdown. We haven't implemented that here yet, if it were my call we'd be doing it once a month. It's a small added expense for the extra protection it provides, although it's hard to get the bean counters to understand this, they just see it as an unnecessary waste of money they'd rather spend on an extra piece of art in the conference room or something."

BTW..... Happy New Year everybody, wishing you reliable electrons all around !

Rgds, D.

 

[ultrarunner]

Yep... similar here on a smaller scale.

For 28 years I used the building load to provide proper Emergency Generator Load for the required hospitals checks... I have also been dipping the diesel tanks since day one.

Last year a new Federal Inspector came out and wrote me up for not having a third party contract for Load Banking and Fuel Testing... Now that we are part of a large organization I was told not to submit a rebuttal citing code as to why we are compliant...

So I go ahead and contact the manufacturer local office and pay $2500 for load banking and fuel sampling...

By the way... I always tested with transfer switch activation every 30 days... this I was told was no longer necessary as we are now load banking...

This turned out to be a disaster... the load bank and generator inspection went well... everything was found in order and the report commented on the 1995 unit was well maintained... so all is good?

Well, the next week I went to do my test and all **** broke loose... the tech reversed the phase when reconnecting so when the generator started it slammed the entire Life Safety and Critical loads... the large HVAC motors hit with reverse polarity... it sounded like a jet engine in the operating rooms...

Over the next couple of weeks I had to replace some large GE Marathon motors... ones that were running at that time...

All I could think and the real crime is we went a week without standby power... had this been a real power outage systems would have failed...

Oh... my dip test no longer meets the requirement for lab analysis... it has been several months and several times the company has been back out because they keep losing the sample collected...

Don't get me started on batteries... I would check the batteries weekly... specific gravity and load test... well, corp said to replace with sealed maintenance free batteries because specific gravity testing no longer required...

This remains a sore subject with me... and I did get reimbursed for some of the damage caused...