A very good four–part series: “Risk and Security in the Telecommunications Industry.”
A very good four–part series: “Risk and Security in the Telecommunications Industry.”
BF Skinner • December 16, 2009 7:27 AM
Attacks on the US beachheads were a major plot point of Richard Clark’s Breakpoint.
I was on the 1/9 train in NYC when the first WTC bombing happened. At the time I thought “stupid terrorist to think they could knock over a WTC building with a single bomb.”
But as the days went forward their node analysis seemed more canny.
Even if they didn’t drop the building and only killed a handful of people.
ALL broadcast television from WTC was off air. That left a single (Public) television station transmitting from the Empire State Building.
Cable was unaffected. But recall NYC has a very resistant to change infra-structure (we didn’t have digital services (callerID, 3way etc) until 96. Cable in NYC was not commonly avaliable until the late 90s.
Many companies were forced to shift their operations out of WTC to other office buildings. Network engineers were being paid 10k’s a week to go without sleep and get the networks up and running again. I didn’t understand why until one of my buddies at Goldman pointed out they were losing 10’s of millions every day they were offline. Some financial firms didn’t survive.
Pretty good shot I’m thinking now.
Alternate processing facilities used to be acceptable if they were more than 50 miles apart. Then on seperate power grids. Are we now looking for continental separation?
J Jacobs • December 16, 2009 9:06 AM
Ignoring the everyday impact of a large external event; flood, fire, earthquake, criminal act (notice the lack of the T word); I wonder how dependent the response plan(s) to such an events are on the same infrastructure that will almost certainly be disrupted…
As has been said many times here, the return on an investment in response planning is almost always significantly higher than an investment in attempting to “harden” the system.
Clive Robinson • December 16, 2009 9:17 AM
Why do people always think of Arni’s state.
Alister MacLean had it with “Good By California” and several other stories since.
Most Geo Scientists I have talked to say something along the lines of “yeah Pacific rim is active, which is why it’s not heavily populated”
If you ask why they don’t appear overly worried you get something along the lines of “It’s so active things realy don’t build up to much”.
When you push a little harder they say “how much damage do you think a small mountin dropping in the sea will do?”.
Then they get a little technical about slopes gradients etc etc.
Then they Mention a couple of islands of reasonable size over on the East Atlantic and they say well “about 50% of this land mass will slide into the Atlantic in the not to distant future and five hours later it’s Good Bye East USA, then seven hours after that it’s Good Bye UK and Western Europe…”
Then you get into a conversation about what further effects there might be and you get the “well conceivably it could shake up the Pacific rim as well…”
So Worst Case the Bible Belt becomes the USA for a while and the Amish and Mormans get to see the second flood.
Like Global warming there is a lot of argument about the specific effects and if the Pacific could get trigered but not the general case. Oh and the time scale is not known but a high probability of within a few thousand years give or take a few thousand years…
Oh and do you remember a celestial body that droped in, in 20 bits on the back of a gas giant not so long ago well there is another heading our way and it’s going to be passing real real close in just a few years.
If it where to hit then the argument is would there be enough energy to trigger other events…
Oh and of course there are the Solar Flares etc and the new Sun Spot cycle to consider both with power and communications infrastructure.
Hmm where did I put my wood burning camping stove 😉
Carlo Graziani • December 16, 2009 9:18 AM
In an adjoining alternate universe where markets actually function efficiently, the collective risk to the economy from network infrastructure failure was factored into the price of bandwidth.
As a result, large technology research and production investments were made to crank up aggregate satellite bandwidth, and to produce a manyfold increase in redundancy of ground facilities and network links, as well as to guarantee smooth failover and fail-safe, easily-available backups. All of which were unfundable at that level given bandwidth prices that did not properly reflect the risk.
Pity markets don’t actually work in this universe. I like cheap bandwidth, but I like reliable infrastructure better.
latency • December 16, 2009 9:23 AM
One can only assume that markets in that universe were responding to the much faster speed of light. Sadly in this universe I tend to be a limiting factor on use of satellite bandwidth.
Carlo Graziani • December 16, 2009 9:39 AM
In our universe, the speed of light is 300,000 km/s, and geosynchronous orbit is 42,000 km. So light-speed effects on a network leg through a satellite are to produce a little less than 300ms to latency. They have no effect on bandwidth at all.
Not sure what use of the network you’re making that is so affected by a 300ms latency (as opposed to about 120ms on a typical overseas connection) that it couldn’t benefit from a large upgrade in bandwidth. But I’d make that tradeoff in a heartbeat.
latency • December 16, 2009 9:41 AM
I note that the story is about networks for financial transactions, where the latency directly determines profitability for the trader. See recent stories on Goldman Sachs ping trading for details.
Carlo Graziani • December 16, 2009 9:55 AM
There may be some important economic activity for which low latency is crucial. But the essentially parasitic activities of ping traders are certainly not it (in fact, serious discussions of ping traders generally include the possibility of adding latency to trades, as a matter of policy, so as to remove the asymmetric advantage of trading houses over vanilla investors). Letting the financial engineers drive discussions of redundancy design for the network amounts to adding insult to the already considerable injury we owe to them.
In any event, I’m pretty sure even those Masters of the Universe would prefer to trade with their peers abroad over 300ms-latency links than not talk to them at all, in the even of an unscheduled outage. After all, that would be the entire point of hedging outage risk, which is something they claim to understand.
David • December 16, 2009 9:58 AM
@Carlo: Satellite transmission a there-and-back-again lightspeed lag of well over a half second. For a large file transmission, that’s trivial. For any sort of back-and-forth real-time communication, be it VOIP or World of Warcraft, that’s a real pain. For downloading web pages with a whole lot of individual pieces to send HTTP requests for, it’s really annoying (although, to be fair, so are the web pages).
Cassandra • December 16, 2009 10:01 AM
You might like to peruse:
which don’t agree with your painting of a possible tsunami disaster.
latency • December 16, 2009 10:13 AM
If you think that the latency of network links is irrelevant then you are living in a parallel universe. You claim that failure to use satellite links shows a market failure – but low-latency and high-latency bandwidth are not inter-changable. David above has offered some applications – but in general any form of interactivity will require a round trip, and a RTT of over 0.5 seconds is unacceptable in most interactive applications.
You also claimed that bandwidth and latency are not related: “They have no effect on bandwidth at all.” but that claim is highly dependent on what kind of protocol you are running over your link. If the protocol doesn’t guarantee delivery (something like UDP) then it is true that bandwidth is unaffected, but those types of protocols are used most heavily in applications where latency is important, such as video conferencing.
In applications where delivery must be guaranteed there will be some equivalent to TCP running on top. And TCP controls its transmission buffers through acknowledgements of delivery. So for all of the non-latency dependent applications that use TCP: bandwidth is affected by latency.
The market functioned correctly and choose to supply lots of low-latency links over physical cables as opposed to few high-latency links over satellite links, as they were in demand by customers.
Dave • December 16, 2009 10:58 AM
Amateur Radio – the only link during catastrophic failure, is commonly ridiculed as “old”. We’ll see!
BF Skinner • December 16, 2009 11:31 AM
@Dave “Amateur Radio – the only link ”
Yeah here’s the thing you see I haven’t been online since the disaster started. I need about 10 minutes alone with your rig. … 3 minutes would do!
BF Skinner • December 16, 2009 11:48 AM
@Clive “couple of islands of reasonable size over on the East Atlantic ….Amish”
Yeah, it’s why I don’t own land on the beach. N’oreasters and beach erosion is bad enough.
We’ve been worrying on those since the world gave Sumatra the finger in 2004. There is no tsunami warning system to speak of in the atlantic, no general awareness, 80% of our response assets and traffic route capacity are within reach of the waters edge
Add the W Antarctic Ice Sheet while you’re at it. What do you’re geophyscial buds think would happen if half a continent fell in the sea?
The Amish (maybe the Menonites, some) though…they have the right of it. They likely would suffer no massive dislocation to their way of life. maybe profit a bit more for food sales. So I always shop http://www.lehmans.com/ for the holidays.
David • December 16, 2009 12:43 PM
I live in the center part of North America, something like 200m above sea level, and on a really stable chunk of rock. Nothing to worry about here aside from blizzards, tornadoes, and the occasional supervolcano.
Methuselah • December 16, 2009 1:25 PM
That supervolcano was nothing to sneeze at ya’ whippersnapper AND it is still active.
But the ice age glaciers now…2 miles and more high…they’ll be back anytime.
Les • December 16, 2009 1:54 PM
Did the movie plot contest start early this year?
DC • December 16, 2009 5:14 PM
Yeah, looks like it. But any good engineer hopes for the best while planning for a spectrum of the worst. Until that “worst” happens in some degree, you look like an idiot for requesting the resources to prepare. But a responsible engineer will do at least some prep and speculation about what to prepare for.
And to build a good plan-tree, you have to have a little fun speculating what indeed could the worst be.
And it becomes a spectrum — some things more or less likely but more or less bad that could happen.
And, rare events do happen in a big universe.
At some point any idea of cost effectiveness (planning for end of world? Why bother!) goes out the window, and a sane engineer stops there. But hey, we all like to have a little fun, too.
Clive Robinson • December 16, 2009 6:10 PM
‘Bill McGuire (UCL)is sticking to the predictions his team have made. Making no apology for backing a worst-case model, he says: “There’s no question of hiding things. If you’re planning for any future disaster you’re not going to consider the least disastrous scenario, you’re going to consider the most.”‘
It was a very nice young lady from UCL who showed me the figures.
And the argument is it going to be “one brick” or “bits of a brick”. The geology sugests the falt line is a shear across at a reasonably steap angle.
So yes the Southampton bods might be right that it has in the past been “more likely” to be bits, but sooner or later that lump is going to slide.
And there is evidence on the US East Coast it has happened before and by somewhat more than the 50m Bill’s team have predicted.
Or to put it another way there where many argument’s in Washington St. About how Mnt St Helens was going to errupt. The bloke who said it was going to blow the way it did sadly died because he was trying to get the evidence to convince people when the bubble burst…
Clive Robinson • December 16, 2009 6:30 PM
@ DC, Les,
Yup you design for the hundred year storm (Brent Spar) and one Xmas it just goes bobing off by it’s self.
Not one of my better projects, to sort bit’s out for…
As they say a one in a million hapens nine times out of ten 😉
dozer • December 18, 2009 8:53 AM
don’t worry, be happy, the sierras will not be part of the chip thats coming loose on the west coast, only the coast range, LA, Malibu etc, Big Sur will be missed, but it may become an island like catalina,
Now is the time to buy beach front property in the central valley and wait for the beach erosion to raise property values.
Subscribe to comments on this entry
Sidebar photo of Bruce Schneier by Joe MacInnis.
Leave a comment