Bottom & links

NSA HQ bldg

NSA campus, Ft. Meade, MD

MINI-TUTORIAL:
THE NATIONAL INFRASTRUCTURES for VOICE and for DATA COMMUNICATIONS
J. I. Nelson, Ph.D.
19July2006; Rev 9Aug06; 27Sept06; 12Mar07
Minor edits/CDRs/ViralRouting  30Jul07; see bottom.

 HOW NSA's WIRETAP SYSTEM WORKS
 
If you just want to know how it works and what it does, it's here.


PREFACE: In 2006, the George W. Bush administration went to great lengths to avoid revealing how a secret warrantless wiretapping system worked.  I  thought anyone with a good background in science and technology could explain it, so I did. Abrupt and disruptive changes in surveillance technology that we must deal with as a society make more sense against the background of the nation's ongoing transition into the digital age, so I have sketched that tumultuous journey here in this backgrounder.
--
Jerry Nelson

SUMMARY

You only get telephone calls out of the telephone system, and it is possible to limit surveillance to the specific "line" of a  specific telephone number.  As the nation moves from this legacy telephone system to the Internet, it becomes impossible to tap phone lines because there aren't any.  Surveillance of  "only a phone call" or "only an email" requires retrieving all the packets performing multiple services for multiple users and parsing out everything they do on line.  A legal justification for one is a technical mandate for all.  

Old laws cannot be "extended" to cover the new technologies.  The technologies themselves are too different.  Just as old telecommunications technologies could not be extended and were overthrown,  so the nation will have to start over with a new approach to surveillance, oversight, and the laws that protect the nation's values and Constitution.

Short of democracy itself, the Internet may be America's greatest gift to the world.  But like atomic power, cloning or stem cells, any great scientific advance bursts upon society showing its faces of good and evil,  demanding we choose between stunning Jekyll and Hyde alternatives.  Surveillance is the Internet's dark side.  The good side is unprecedented wealth and  creativity.  The Internet succeeded where the centrally-controlled, "command economy" of the telephone world failed because the Internet follows a design philosophy of independence between its parts, possesses a culture of openness, and grants political empowerment to its users who do not have to ask anyone's permission to invent outrageously new and successful services.  

INTRODUCTION

Less than two decades have taken us through an upheaval in telecommunications with multi-billion dollar mergers, multi-billion dollar bankruptcies, and extremes in human miscalculation of how to invest money or comply with legal and financial realities.  The disruptive shift from a national telecommunications infrastructure for voice transport to a national telecommunications infrastructure for data transport threatens us with similar miscalculations and disastrous, unintended consequences if we repeat our mistakes and mismanage the  political, legal, and regulatory responses to the same technology.  Uncomprehended power in computation, communication, and the manipulation of information now touches everyone's personal life and our government's power to monitor it.

The legacy national infrastructure for voice communications could not be extended to serve the nation's data communications needs.  The new data transport systems could not be piggybacked onto the legacy voice transport system.  Billions in national treasure and prestige were lost in the ensuing upheaval.  Asian competitors scrambled as some of the finest technology in electronics and photonics went on the block for pennies on the dollar.  Global Crossing's purchase by Singapore Technologies Telemedia Pte (STT) and Hutchison-Wampoa (Hong Kong) could have passed the graphics files loading into the guidance systems of cruise missiles and the real-time battlefield images coming back from unpiloted drones over American troops at war through Asian military powers not necessarily well-aligned with our national interest.  (Hutchison-Wampoa withdrew; STT owns a 71% stake of Global Crossing.)

Another round of mis-steps that later prove devastating for society are likely
if it is assumed that political and judicial safeguards developed for the fading voice transport era can be extended to the domain of data transport.

Mis-steps are likely because society's political and legal response to global telecommunications' upheaval will be crafted by civic leaders who may understand both voice and data systems even less well than did the now-fallen (e.g., convicted and sentenced) titans of those industries.  

To understand the problems civic society faces in crafting a legal framework  for surveillance and for the preservation of constitutional law as we move from one network to the other,  it is necessary to understand the two networks.   A signal that the technologies are not understood is the use of the word "extend".  The call to "extend"  a law crafted for the dominant network of the last century to the disruptively different network of today identifies someone who understands neither.  

This mini-tutorial is written for policy makers and lawyers not deeply familiar with technologies that, like a hurricane over warm water, have quickly grown to open new channels of power and topple barriers we thought protected us.  By putting both old and new networks under the microscope, we also see at a glance what has made the Internet a fountainhead of innovation, 
America's greatest gift to the world short of democratic government itself.  Yet the dark side of the same technology is surveillance.  The new national infrastructure for telecommunications -- the technology -- has already been chosen by technologists, but a fork in the road has not been chosen by society.  The exuberance and renewal of civic society are palpable on one side, and the sad faces of a surveilled society illuminate the darkness on the other.  The technological systems could not be extended, and I urge you not to confirm for yourselves that the legal systems cannot be extended either.  Start over.  

Find out what drove the Internet to greatness and you can insure that the Internet continues to drive society to greatness.  
A piecemeal approach to services and civil liberties will not steer either a great technology or a great society.  Imposing the technologies required for surveillance will redefine the Internet's principles of operation.  20th century "tapping" of  a 21st century integrated services network does not draw off an innocent glass of port, it poisons the barrel and the future.  

If you wish to skip the mini-tutorial, the bottom line is that you can tap only one phone line, but on the Internet you can only tap everything because there are no lines.   Tapping VoIP calls on the Internet ("Voice over Internet Protocol")  logically and technically compels us to extend surveillance to everything everyone does on-line.  



Cable laying ship, Pirelli Systems.     Marine Cable Landing, Pirelli Systems.  

The nation's legacy infrastructure could not make the transition to digital communications.  Meanwhile, advances in photonics created new networks with greater capacity than all prior networks in history combined.  Unfortunately, new companies breaking old market restrictions quickly created multiple such super systems on land and sea and suffered the economic consequences.  The United States lacks an effective national communications infrastructure today, and continues to fall behind other nations in broadband usage.  It is an odd societal outcome when so much of the technology originated here.     (Marine cable landing, Pirelli Systems.)


THE LEGACY VOICE NETWORK

You are now an honorary engineer and we are going to build a phone system.  With 20 million people in New York State and 34 million in California, it would be nice to be able to connect even just a quarter-million of them at once.  The simplest way is a phone circuit  from New York westward, and another circuit back.  The circuit -- the same word as circle -- is a pair of wires so that any current sent to California returns in a system which remains in balance and free of electron pile-ups.  So we need 4 wires for each customer, or 1 million wires in all. Ignoring the plastic insulation, and choosing thin 22 AWG wire (American Wire Gauge 22, a common size for phone wiring),  this cable will weigh 1,944 lbs/foot, a cool ton per foot, and there are 5,280 feet in each mile to California.  

We have had our first hard lesson in telecom: it is necessary to combine many -- thousands, millions -- of customer's traffic streams into one very high speed wire, microwave link (mid-1900s) or glass fiber (Y2K).  How to combine traffic streams and sort them out later is the central question on which legacy voice and ascendant data networks have made fundamentally different choices.  Like the egg's choice of an X- or a Y-bearing sperm, this difference is destiny.

The combining of many customers' traffic streams is called multiplexing; sorting them out later is demultiplexing ("mux" and "demux").  

COMPRESSING TIME:  A simple trick makes multiplexing possible: the fast squirt.  Since the 1970s it has been easy to record your telephone voice for an 8-thousandths of a second and convert it to a digital stream that is easily stored and played back.  In the universe as we know it, the recording of  8-thousandths of a second of your speech will always take an 8-thousandths of a second.  However, we can play the snippet of speech back 32,256 times faster than we recorded it -- the fast squirt.  And, as we said, these data bits  (digitized voice) are easy to store, so we can store the same speech snippets from 32,256 paying customers and squirt them all into our wire in the time it took to record a single one of them.  And that's a good thing, because, as soon as that 8-thousandths of a second is over, those 32,256 paying subscribers are ready as a group for their  next fast squirt.  The next 32,256 samples are buffered (stored and queued) and must be shipped out the door.  Done!  Obviously we have the start of a powerful multiplexing scheme here.  Voice samples with these kinds of numbers were shipped out on a fiber optic cable typical of MCI's long distance network in 1998, as last century's technology neared its peak.  

Looking ahead to the other end of the line, your speech plays back smoothly.  By the time we have (slowly) played one 8-thousandths of a second of your speech, the next 8-thousandths is ready. The jagged and abrupt way in which we obtained each sample remains unnoticed, provided we string the samples back together without interruption.  

The number "8-thousandths" is not interesting, provided every country in the world agrees to build every piece of telecom gear with a pacer that clocks in at this speed.  They have. The other numbers are more important because you pay for them.  Many readers may have purchased a "T1" line or may have been told their Internet access has "T1 speeds".  Telecom multiplexing hierarchy level number one ("T1") was achieved in 1962, packing 24 phone calls (not 32,256) onto a wire running at 1.544 megabits/sec or 1.544 megaHertz.  The T3 level multiplexed 672 phone calls into a wire running at 45 megabits/sec or 45 megaHertz.  The frequency or  bandwidth numbers (1.544 or 45 megabits/sec) have survived in the marketplace even if the cables now carry data, not voice calls.  At higher levels of multiplexing, the multiplexed signal is carried by a glass fiber.   129,024 phone calls per multiplexed channel is common; bragging rights start at four times that (Optical Carrier Level 768), and after that the telephone companies' multiplexing game is over.  The torch passes from electronics to photonics,  to the trick of tuning 64 lasers to slightly different wavelengths ("colors" the journalists call them) and using optics to separate the 64 channels at the other end.  


TIME DIVIDED INTO FIXED SLOTS:  Let us look more closely at one 8-thousandths of a second.  In this time, 
32,256 paying subscribers are lined up to get their 8-thousandths of a second of speech shipped out before the clock ticks again.  During this crucial 8-thousandths of a second, each customer in turn gets exclusive use of the entire optical cable, but only for 1/32,256 of that 8-thousandths of a second.  In that time you get to transmit 8 bits (not 8 million for a photo, but eight, period).  It is a straight jacket: 8 thousand times a second you get exactly 8 bits, never any more, and, even if you are not talking, never any less.  While the network has a fabulous 2.488 gigaHertz cable, you only get to put 8 bits onto it 8000 times a second -- 64 kilobits per second in all.  That is why dial-up modems never do better than 64 kilobits/sec.  

The slot assignments also never change.  If you are customer 32,156, you will be 32,156 on that telephone company switch until you hang up the phone.   Obviously we have the start of a powerful scheme for DEmultiplexing here.  The 8-thousandths of a second interval is rigidly divided into 32,256 "time slots" that never change, and one slot is assigned to an individual customer, and for the duration of his phone call, that assignment never changes either.  Asking for slot 32,156 gets you the smooth sound of one person's telephone call.  


TIME-DIVISION MULTIPLEXING:   The fixed-time-slot approach is called
Time-Division Multiplexing (TDM). The slots into which we multiplex many customers are made by dividing an instant in time that is 1/8,000th of a second, and is the same the world over.  Companies with equipment that divides faster can make more slots and more money.  

The choice of Time-Division Multiplexing for the nation's telecom infrastructure had tumultuous consequences for the phone companies beginning in the 1990s,  as the tide of data traffic rose and the fatal flaw of the nation's voice infrastructure emerged (the customer never gets more than 64 kilobits per second and the phone company has to tie up the infrastructure for that slot even when the customer puts no traffic into it). The decision to use Time Division Multiplexing is now having profound consequences for the evolution of surveillance and constitutional law in civic society, because it preserved the fiction of a phone "line" which could be "tapped". `  Such virtual "lines" are not hard to understand.


5ESS Switch -courtesy SNiPLiNK-LLC-PennsaukenNJ
5ESS switch (right; courtesy of SNiPLiNK, Pennsauken NJ)
A switch is just a computer system with hard drives and
a lot more input/output jacks than a single port for a printer.

MAKING A RESERVATION:   A quick phone call from New York to LA finishes the TDM mini-tutorial.   We consider only the moment of silence between the last digit you dial and the first response (ring or busy) you hear.  Within 18,000 feet of your house as the wire lies is a telephone company's central office building  with a time-division multiplexing switch of the sort we have been examining.  Your voice is digitized there and, as soon as the switch assigns you a slot, it lines up the next switch, probably still in your city but no longer in your neighborhood.  That bigger switch may achieve more than 32,256 slots per cable and there's no telling what number your slot will be, but, until all the switches from New York to LA have each committed a slot to you, the call has not been placed  (has not been set up; later it must be torn down).  It doesn't matter what the next slot numbers are, as long as you get one.  Each switch on the route knows what slot number to find you on in the  incoming stream of traffic, and tells the next switch what slot number it has has put you on in its own  output stream.  The last switch on the route has an actual copper line to the party you are calling, connects electrically to it, and finds out that it is either off the hook or ringable.  Your call has gone through, and the switch in LA tells your local switch in NY what outcome it should signal to you (busy or ringing).  

One switch lines up the next to reserve a new slot for a new phone call using Signaling System 7 (SS7).  SS7 is a large set of commands that includes the ones  NSA uses to open a new phone tap by remote control.  A lot of the SS7 commands used to be sent with Touch-Tone sounds, one of which could be imitated by a child's whistle given away inside boxes of Cap'n Crunch cereal (http://www.webcrunchers.com/origins.html).   By taking control of switch reservation signaling, phone phreaks competed to get a call stitched together around the world, as evidenced by the enormous delay required for the winning phreak's voice to come out another phone in the same room.  Today, SS7 signaling is not done with Touch-Tones, and is often carried on a separate, secure network used just for signaling.

VIRTUAL CIRCUITS FOR PARTICULAR CALLS:  Nothing more changes after you have your connection.  The reserved slots remain the same and they are all yours.  The slot numbers and even the choice of cities in the city-to-city hops may have been quasi-random choices, but, because they are fixed and they are yours once chosen, we may say that each subscriber of a completed connection owns a single, specifiable "virtual circuit".  It is not a physical circuit bolted down in current-carrying copper, but it is a stable, measurable piece of the national telecommunications infrastructure that has been assigned exclusively to a particular subscriber.  

In the voice system, each customer of a legacy Bell telephone company has a virtual phone line that the National Security Agency can tap.

No one needs to climb a phone pole.  The idea of "tapping" (as in tapping a barrel to let some contents escape or touching copper wires to tap off some current)  is quaint.  The National Security Agency can signal the switch -- typically the first one, the switch in your neighborhood -- to always make two (or more) "next switch" reservations for your slot.  One begins the chain of switches needed to build your virtual circuit to the number you dialed, and the other begins a chain of switches leading to the National Security Agency's computer rooms in Fort Meade, Maryland.   Every call you make now goes to two places (or more).  The NSA can also ask to share your Caller ID signaling so that its sophisticated call-monitoring resources need only "listen to" (analyze) your calls when certain people check in with you by phone.  You will hear nothing on your line,  your calls will not take longer to complete, and you won't be billed for the extra long distance connection.

CALL DETAIL RECORDS   

INTRODUCTION: The Call Detail Record (CDR) is the written reservation for a single call.  Because little about Call Detail Records is generally known, a longer treatment seemed in order.  If you need to skip it, the bottom line is that, as a call is begun (you lift the receiver) and "reserved" (the call is set up after you dial the last number), a timestamp is recorded for each step of the process, along with a record of both the called and calling numbers, the billing number if different, the long-distance carrier used by you the customer, additional items if you put someone on hold and set up a three-way conference, and outcome codes.  If the call wasn't successfully answered, there will be no bill later.   Subtracting one timestamp from another gives the duration needed for billing, so CDRs get run through lots of "billing software" later to look for completion codes that mean "success", to perform these subtractions and tally minutes.  The National Security Agency likes to get these CDRs plus all the incoming and unsuccessful and non-billable (local) call CDRs as well -- extra CDRs that phone companies have no business case (financial reason) to archive and sometimes don't even generate.  Congress has created an incentive for NSA to flag phones as tapped because only tapped phones generate all the extra CDRs, and once tapped,  the phone companies are legally  required  to deliver all the calling records to the government.  Now the details. 

DETAILS ABOUT CALL DETAILS YOU NEVER WANTED TO KNOW: The Call Detail Records (CDRs) that telephone companies generate for commercial purposes on untapped phones (and then turn over to the National Security Agency surreptitiously  (USA Today 11 May 2006: http://www.usatoday.com/news/washington/2006-05-10-nsa_x.htm ) are very different from the CDRs that telephone companies generate on phones that have had their taps turned on (which are then turned over to the FBI openly, as required by Congressionally-mandated surveillance rules stipulated in "CALEA", the Communications Assistance for Law Enforcement Act, 1994).  "Turned over openly" means the FBI does not have to deny that it is doing what it is doing when it refuses to tell you what it is doing. 

For an introduction to CALEA, see: http://en.wikipedia.org/wiki/Communications_Assistance_for_Law_Enforcement_Act

For lawyers seeking the law itself, see: 
http://www.gpo.gov/fdsys/pkg/USCODE-2011-title47/html/USCODE-2011-title47-chap9-subchapI.htm
Better formatting:
 https://en.wikisource.org/wiki/Communications_Assistance_for_Law_Enforcement_Act

For telco executives concerned they still haven't installed enough spying technology, check with the FBI at their CALEA compliance site,  http://www.askcalea.net/

We consider the classic Call Detail Record first, from the days when most phones were not tapped, and 90% of taps were for calling pattern data (phone numbers) only, not the voices that NSA now analyzes with voice recognition computers. 

MAKING A CALL DETAIL RECORD:  Before your call's route to Los Angeles was reserved, the switch in your local central office (the first in the chain), queried a database to find out your personal choice of long distance carrier, and then began the switch reservations accordingly.  It also appended a new record, the "Call Detail Record" (CDR), to a database on a different server, and recorded the number you dialed in LA, the long distance carrier specified, and the result of the attempt, including the start and stop times (the timestamps) from which your billable minutes will be derived if the result code is "successful".  In the old days, the switch itself stored and minded the database.  

The switches at both ends of the call must communicate.   LA told NY whether to signal you "busy" or "ringing" at the outset of your call (at the end of their reservation process), and, at the end of the call, either party could hang up first, so either switch must be able to initiate information exchange with the other to begin "call tear-down".  Because both switches are activated and each is responsible for keeping track of charges to a paying customer, both generate a Call Detail Record; thus, CDRs come in pairs.  Because the switches must communicate, each CDR can (and does) list both the called and calling number.

THE STORAGE OF CALL DETAIL RECORDS:  In many independent phone companies, the Call Detail Records (CDRs) are read off the switch that set up and tore down the calls.  The CDRs are read by bill extraction-and-presentment companies at least once a month.  Billing companies have codes to access the switches remotely and request a dump of billing data.  The phone company (landline or cellular) that owns the switches eventually moves the CDRs off the switches to make room for new ones.  

Records are consolidated onto large, central databases and kept for several years.  Some companies keep the raw Call Detail Records; others, the more compact billing records derived from them.  The derived billing records bear a one-to-one relationship with the full, raw Call Detail Records -- each is a record of a single call.  In industry parlance, the derived billing records are often called "Call Detail Records" insofar as each describes details of a single call placed by a customer.  However, the more compact, derived billing records typically have no information about incoming calls, and sometimes only the called number is listed and the identity of the caller (the calling number) is implicit in the file name or document title ("Services rendered to CustomerXYZ").  There is no progression of time stamps as the call progresses through setup and teardown, only the billable duration and start time.

MCI is famous for keeping all the raw Call Detail Record data and mining it for marketing purposes -- customer leads;  up-selling active business customers;  cross-selling consumers on novel dialing plans such as "Friends and Family", and for corporate strategic planning (infrastructure investment).  

The Bell System has more capital equipment than independent telcos do.  An accounting system (computer server running a database management system) may be set up separately from the calling system (5ESS electronic switch).   The data for  all successful outgoing billable calls are transferred to the accounting server as they occur, and the urgency for bill extraction companies to harvest (empty) the switch before it overflows disappears.  In large cities, citizens whose phones are currently flagged for tapping get their call data placed on a separate server, not necessarily because there are so many people in this group, but because the machine holding these people's information needs to be networked into government computers.  Phone companies would rather build and interface special systems with government systems than contend with government browsing of their entire infrastructure.  

Customers of legacy Bell telephone companies generate a history of all people who called them, and whom they called or even just attempted to call, in Call Detail Records that are accumulated in searchable databases in either raw or reduced form.  These databases  can be shared with the National Security Agency.


CALL DETAIL RECORD (CDR) for ONE PHONE CALL

field #bytes description
Serial# 4 .
Correlation ID byte 1 Also called the "Correlation Group"
Protocol ID 1 .
Network indicator 1 National or international, 14- or 24-bit SS7 point codes (addresses)?
Data Length 2 .
Fixed data variable depends on Protocol ID field
Optional data variable typically an 8-byte timestamp & a 2-byte message
OPC 4 Origin Point Code (of  SS7 signaling chnl + hdw controlling the call)
DPC 4 Destination Point Code
Trunk ID 2 .
Called Number 28 .
Calling Number 28 .
Calling Party Category 1 ordinary, payphone, operator
CIC 4 Carrier ID Code; your long distance provider
Release Cause Indicator 2 What command ended call?
Reason (ended under what conditions?) 2 0 - No Error
1 - Timeout
2 - Sequencing error
3 - Reset Circuit (backhoe?)
-
field #bytes description
IAM Timestamp 8 Initial Address Message
Monitor ID for IAM 2 .
Link # for IAM 1 .
ACM Timestamp 8 Address Complete Message
Monitor ID for ACM 2 send "bkwds" to certify
Link # for ACM 1 that virtual ckt is complete.
ANM Timestamp 8 ISUP ANswer Message
Monitor ID for ANM 2 from called pty switch;
Link # for ANM 1 commences billing time.
REL Timestamp 8 RELease Message
Monitor ID for REL 2 Hangup; can come 
Link # for REL 1 from either party.
RLC Timestamp 8 ReLease Complete
Monitor ID for RLC 2 Can be "normal" or "busy" and
Link # for RLC 1 usually ends billing.



- NOTES:

1. This basic CDR is about 150 bytes long.  The CDR is a computer log file whose length can grow.  Pressing HOLD and adding a 3rd party to your call lengthens your call's data record. 

2. CDRs record the steps taken by the telephone switch in your neighborhood Central Office to create and end one phone call for you.  Major phone companies generate between 150 terabytes and 1 petabyte each of CDRs per year (2005).

3. CDRs are written in pairs to disk drives in the switch setting up the call and in the switch serving the called party.  Phone companies avoid CDR duplication, but FCC survey data for the nation as a whole can be off by a factor approaching 2x.

4. Each of the messages whose occurrence is time-stamped here is itself a multi-byte entity of rigidly-specified format.  The specifications as a whole are said to fill a 1.5 cu.ft. carton and the mostly C-language Signaling System 7 (SS7) program that executes them is about 10 million lines of code, but most are concerned with graceful recovery from exceptions. 

5. Additional signaling to support 800 number translation, 911 calls, caller ID etc.generates additional messages not time-stamped in the CDR. 

6. "ISUP" (ISDN User Part) is a collective name for commands used to set up subscriber requested trunk calls & has nothing in particular to do with the "Integrated Services Digital Network." 

Corrections/additions welcome at jerry-va curlysign speakeasy dot net.

Attribution Jerry Nelson Sept 2006
revisions:




Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 License.



TENSIONS BETWEEN COMMERCIAL vs. SURVEILLANCE NEEDS for CDRs:  The Call  Detail Record (CDR) databases change rapidly, and would have to be shared with the NSA on an on-going contractual basis.  MCI alone adds 90 terabytes of customer records per month (2005) -- mostly Call Detail Records.  This is roughly a petabyte (1,000 terabytes) per year, requiring a new, $4 million EMC Symmetrix DMX-3 storage system each year (9 cabinets, 2,400 disk drives).  With some companies flushing their switches' calling data into other servers as they arise while others leave Call Detail Records on their switches for a month, it is technically feasible but administratively all but impossible for the NSA to collect Call Detail Records from the nation's 25,000 voice switches by itself.  The NSA is all but forced to let each phone company consolidate its data first, and then ship it on an ongoing, contractual basis to the National Security Agency.

There are tensions between market and government needs.  Commercially, only (potentially) billable calls merit creation of Call Detail Records and, until CALEA was passed in 1994, telephone companies did not create or keep Call Detail Records for non-toll calls within Local Access and Transport Areas (roughly, within one area code) even though the switches could generate them.  Bill extraction-and-presentment companies do not care about incoming calls and sometimes the phone companies do not store that information either, but the NSA wants it.  Frantic attempts of one person to reach another by phone at critical times are of interest to NSA even if the call was never answered or even completed, cannot be billed, and CDR timestamps would not normally be stored.  The NSA would prefer to have the richer information of the true Call Detail Records as written by the switches, not the reduced version produced for billing purposes that is cheaper to archive.  Phone companies would want to be well-compensated for any effort, expense, and work not justified commercially. 

THE AT&T "DAYTONA" DATABASE of CDRs:  Long distance companies are in a much stronger position than the local phone companies to give NSA everything it wants.  Because a long distance company has no local lines, it must pay other companies to deliver (complete) their calls for them.  Call completion incurs "settlement charges" owed to (potentially) any extant local phone company in the country.  A central database is needed to generate and control settlement charges.  These databases are usually kept in a central part of the country (Kansas for AT&T), so the NSA's national data collection-and-consolidation problem is solved.  Better still, AT&T updates their database in realtime.

If the long distance provider you signed up with is AT&T, then, 5 seconds after you hang up with Grandma your call information is in Kansas, and, by evening, your information is on an NSA computer in Fort Meade, MD (see the Electronic Frontier Foundation lawsuit filed 31 Jan 2006, https://www.eff.org/nsa/hepting ).   In a more transparent society, citizens could insist that, with a budget of $4 to $5 billion a year not counting satellite costs, the National Security Agency should perform their updates in real-time just like the phone company.  Then, since government computers are far more secure than AT&T's, the NSA should charge AT&T to access the government for data on their customers and save taxpayers some money.  

A TAPPED PHONE GIVES NSA EVERYTHING IT WANTS:  If you have "call waiting" or "caller ID" service, appropriate flags are set in your "service profile" at your neighborhood Central Office.  You are billed for it.  If your phone is tapped, another flag in the same service profile is set, and the rest is automatic.  
The good news is that, as far as we know, the phone company has never made a mistake on a customer's bill and charged him for the added service of tapping his phone.  

When a phone is flagged to be tapped, the phone company, seemingly out of the goodness of its heart and with no compensation from the government, voluntarily saves to its billing database information -- additional information which generates no billing and is normally discarded, including:
Normally, local calls would be neither recorded (switch doesn't waste time generating a CDR), nor saved for later billing (perform transactions with a database server and take up space on its storage drives).   Setting the wiretapping services flag "ON" creates a lot of unbillable "billing data".  These are the most complete Call Detail Records in the business.

Bell companies typically set up a separate database server to keep all the data for all their customers who are currently under  government surveillance.  

The database server dedicated to the calling records of a telephone company's government-surveilled customers places a call of its own, and then uploads this data into the FBI's or the National Security Agency's databases. 

It pays the NSA to set the phone tap flag to "ON".  Problems and shortcomings arise when the National Security Agency must  obtain Call Detail Record databases retroactively, instead of on-line as they are created.  The problem is that the NSA must go in secret to a public corporation, and the shortcoming is that not all the CDRs are there.  It is easier and less unpleasant for everyone concerned when the phones are flagged to be tapped.  If the service flag for "wiretapping services" is set "ON", not only are all Call Detail Records saved, but your local Central Office switch will now be set (automatically re-programmed) to generate CDRs that were never even generated before, such as records of your failed, incoming and local calls.  Finally, the NSA's need for secret meetings to pressure senior management to turn over data evaporates,  because Federal law (CALEA, 1994) makes it not only legal, but legally required that the phone company export the data directly into law enforcement agency databases.

These differences in CDR generation (more complete) and delivery (they do it for you) are an incentive to NSA to tap phones. 
Congress forced the telecommunications industry to install phone tapping technology for law enforcement agencies (the CALEA Act, 1994, "Communications Assistance for Law Enforcement").    Today, it is likely that NSA plows through the nation's phone books, setting service profile flags for tapping services to "ON".  Lost in the shuffle is the fact that the National Security Agency is not a law enforcement agency, and that the phone taps are not authorized by any warrant.  CALEA was a strong foundation.  A technological edifice has been built on it.   

In Silicon Valley, they laugh at the saying, "Build it and they will come" because everyone knows that great technology often  requires great marketing to achieve success.  Government is different. 
CALEA was a strong foundation, and a technological edifice evolved on top of it.   With the CALEA law enforcement act, Congress made it possible for a gumshoe to log onto a computer system and turn on a phone tap without a lot of back-and-forth correspondence with telephone company employees.  The guy is overworked, it was nice to help him out.  The NSA came in and drove the system with computers to thousands of taps an hour based on billions of phone records a year.  

The "give an inch, take a mile" pattern is now repeating itself with geolocation.  Congress required the telecommunications industry to put Global Positioning Satellite chips or other position triangulation technology into every cellphone so that a dazed or crazed caller would not have to describe her position on a 911 emergency call.  Nice to help her out.  This real-time data is also stored with cellphone Call Detail Records.  The government won a lawsuit to gain nearly unfettered access to databases recording every citizen's physical location for every cellphone call they make, not just 911 calls. Since your physical tracking data  was stored on the billing database, the court's argument was that tracking was only billing data.  

We have looked at calling histories.  The routing of phone call voice traffic  to NSA voice recognition computers was not covered; in effect, it is just a 3-way conference call.  The investment and specialized knowledge needed to change the SS7-related firmware of an 5ESS switch, and the history of unwanted consequences from software changes, suggest that Call Detail Record time stamps for connections to the National Security Agency were never falsified and still exist for all Americans whose phone conversations were forwarded to the government as they were placed.  There will be smaller phone companies serving cities with Arab-American immigrant communities whose Call Detail Records document citizen surveillance. 

This review of Call Detail Records completes our portrait of the legacy phone system in its full maturity, and we turn to the new kid on the block, the Internet. 

This is the Mini-Tutorial on Telecom Networks & the Internet.
--=o=--

As the last century ended, the nation that gave the world the Internet now created the search engine, the eBay auction, on-line music, as well as experiments that failed in the dot-com bust of 2001.  More is coming.  The legacy telecommunications infrastructure could not accompany the nation on its new adventure.  Such innovation, such change! What were the voice networks missing that the data networks had? 


THE NATION'S  NEED FOR NEW DATA NETWORKS

Data networking started earlier in our country than in the rest of the world.  We invented the isolated transistor; we invented the integrated circuit and the semiconductor industry that learned to put thousands and millions of transistors on one integrated circuit chip; we used the chips to invent the PC, and we put a PC on every desk.  

PCs on every desktop made a communications revolution necessary, but companies in the Bell System would rent only empty lines, forcing  corporations to turn elsewhere for equipment to provision them.  The data router was invented, new companies like Cisco were born, and the United States owned the data equipment industry even before the Internet became prominent.  Single corporations built data networks whose geographical reach and capacity exceeded the national telephone infrastructure of entire nations.  The invention of the World Wide Web (1991; brought under the aegis of the World Wide Web Consortium in 1994) made  data networks valuable to consumers, who helped drive the World Wide Web from 2,000 sites in 1994 to 20 million 6 years later.

As data traffic grew both at home and at work, weaknesses emerged in a national telecom infrastructure based on voice transport. The legacy voice operators offered data services that  were mostly forgettable:  DDS (Dataphone Digital Service, 1974) for example, and SMDS (Switched Multimegabit Data Service). Something was wrong. 

The nation's telecommunications infrastructure could never make the transition from voice to data.  It failed by design.

1.  RESERVED BANDWIDTH HURTS TELECOM PROVIDERS and THEIR SUBSCRIBERS

Assuming a spoken conversation would occur, it was reasonable to reserve a small, fixed amount of bandwidth for it. But computers demand 
enormously more bandwidth than many phone circuits one instant and none the next.  The telephone system could neither satisfy bursts in demand nor staunch the waste when waiting in silence.

By its design, the legacy telecommunications infrastructure could not morph into a broadband network for data.  Men leading an industry with $100B in local  and $80B in long-distance telephone revenues realized that other men were building new networks that had greater capacity than theirs, indeed greater capacity than all prior networks built in history.  The growing volumes of data traffic and data capacity on other networks would trivialize cash flows derived from voice.  There was no time for the technological response needed to give the nation a broad-band infrastructure designed from the ground up for data.   Bell system corporations turned increasingly to legislative and legal moves against purveyors of change.  With the fiber optic "champions of change" each fielding networks with greater capacity than all prior networks in history, this group soon found itself in financial troubles as existentially threatening as the technical troubles then devastating their legacy competitors.  The nation as a whole floundered.  

2.  CENTRALIZED CONTROL & COMPLEXITY RAISE COSTS

Lockheed Martin's F-35 needs 1.1 million lines of code for flight control (5.6 million lines for the whole plane), but a it takes 18 million lines of code to run a Bell System 5ESS switch.  The Bell switch make reservations on the next switch with an equally complex SS7 signaling and control language.  No inventor can expect a corporation running such a network to "just try something."  

There are custodians of the code (Bellcore, now Telcordia, manages 100 million lines of software instructions for use by telecom companies), but problems still arise.  Companies have their own separate systems, one each for billing, for customer relationship management, operations, administration, maintenance and service provisioning. Companies are acquired; employees leave.  Regional operating companies running on over 200 million lines of code may find there is no one left who can understand, maintain or modify large sections of it.  

3. FAILURES HAPPENED

A switch crashed on 15 January 1990 and faithfully sent an out-of-service notice to its neighboring switches so that they would know to make reservations (place calls) around it.  When it turned out that trying to respond to an out-of-service notice would itself cause a switch to fail, the ripple of  failure did not stop spreading outward until 60 thousand subscribers had dead phones and 70 million incoming calls were never completed.  In June and July of 1991, major failures occurred in Baltimore (spreading to Washington DC and Northern Virginia), in  LA, and in  Pittsburgh, where 1 million customers were affected for 6 hours. An error in the SS7 signaling software caused the lines over which one switch requests a reservation from another to become jammed.  Two months later, a hardware failure took out service to 10 million customers in New York City--and three airports--at 10:10 AM.  A comedy of errors prevented power restoration until  evening.  The airports did not clear until midnight, and many passengers could not call out to report their plight.

The legacy voice network lacks powerful principles of separation permitting one domain of the national infrastructure to be enhanced without unintended side effects on others.  (There are similar issues in the national infrastructure for AC power distribution.)  Anyone inside or outside the telephone system wishing to make changes confronts many millions of lines of computer code that is proprietary, locked inside corporations, and itself so interlocked  that a small change one place can produce disasters in another.  Again: no entrepreneur ever asked the custodians of such a system to "just try something" and got "Yes" for an answer.  

4. INNOVATION FAILED

The reality of this complexity has resulted in a cautious approach to innovation.

Twelve years after the first primitive 911 service was launched in  Haleyville, Alabama in January 1968, the development of today's modern 911 system was complete and one system went on the air (Orange County, Florida, January 1980).   The President signed a bill declaring that 911 should be available nationwide in 1999.  Rural counties are still working to implement the complete feature set today, 39 years later (2007).  

Ordinary citizens can use the Internet to create new services (eBay, PayPal), but only authorized phone technicians can get their hands on the phone system.  What new businesses or innovations have those technicians created?

There is Speed Dialing, where you can dial [star] 7  5  [2-digit code] [pound] instead of a 7-digit number ($4/month).  You must contact a technician to enter your list of numbers into the system for you.  You cannot save the company and yourself money by using a Website to print and maintain your list at your convenience.  

When counting the number of new commands that the phone system has added,  remember that it takes one command (dial a special number, then a special 2-digit code) to turn on a distinctive ring if Grandma ever remembers to use the special number you gave her, another command to cancel the service, and a third command for people who can't remember the first two and would prefer a single command that just toggles the service on and off.  Wake Up Calls are $2 each and $4 for business accounts, so don't sleep over on the office couch, it's expensive.

Another new service of the Advanced Intelligent Network (AIN) is "Call Trace (Customer Originated)".  Ordinary citizens may  automatically trace their last incoming call.  You just dial *57, listen to the message and then dial 1.  If the call could be traced, you will hear a message that it has been traced. Traced call information is provided upon subpoena to law enforcement officials, but not to the customer. (Citel, Inc.; see also Verizon.)

In contrast to the national telecommunications infrastructure for voice, the 
World Wide Web was invented by a single individual in 1991, standardized by users --  not service providers -- and grew to 88 million sites around the world and  25 billion pages indexed by Google in 15 years.  Nothing in 100 years of voice telephony matches this record of success.

5.  STILL ANALOG AFTER ALL THESE YEARS

To sign up subscribers and drive revenue, a telephone company must multiplex.  To multiplex, the telephone company must digitize.  The national telecommunications infrastructure began digitizing voice in the 1970s, following the start of T1 trunk multiplexing in 1962.  

Thirty-five years later, the corporation is digital but the customer has been left with analog access.  Customers do digital things like use PCs and send faxes.  The fax machine's digital scan of every page must be encoded as analog sounds to go over a voice channel.  At the first telephone company switch it hits (the 5ESS switch in your neighborhood), the machine's voice sounds are digitized.  We digitize  the analog encoding of a signal that was digital to begin with.   At the last telephone switch on the journey to the dialed destination, the digital signal is converted back to analog sounds to go over the phone line into which the destination fax machine is plugged.  The fax machine makes the final analog to digital conversion, and puts pixels on the page.  

Faxes are digital, and, to send one, we as a nation make four digital/analog conversions, all of them unnecessary.  Digital communications between computers (e.g., an e-mail) are scarcely different.  The game is digital to analog to digital to analog to digital.  The  score is: Conversions, 4; National Digital Infrastructure, 0.   A national telecommunications  infrastructure for digital communications has never been completed.  

A chain is only as strong as its weakest link, and a system is only as fast as its slowest component. The analog connection between a customer’s home computer equipment and the telephone company’s Central Office is the slowest component of the nation’s entire telecommunications infrastructure.    

DSL modems and dial-up modems both make the same conversions that  fax machines do.  The only difference is that, after hitting the first central office, much of the subsequent long-distance journey is taken off the national voice infrastructure, whose switches are designed to make fixed bandwidth reservations that are too expensive and inefficient for data transport.  Data transport is performed by companies less well known to the public, such as Broadwing or Level(3) (now merged), since the networks of phone companies have lacked the national footprint and/or sufficient capacity for data traffic.  Data traffic surpassed voice traffic sometime in 2000/2001, roaring past at an annual growth rate approaching 100% a year and, by 2005, was about 4 times greater than the voice traffic volumes for which the networks of the nation's telephone companies   were designed and built.


Telecom-DataOvertakesVoice  

Internet users are politically empowered to innovate.  Traffic explodes for them, not voice.  
From  http://www.witbd.org/articles/digital_communications.htm



6. BIG SHIPS TURN SLOWLY

The legacy voice industry made one more move as the data revolution matured: ISDN, the Integrated Services Digital Network.  The Time-Division Multiplexed, fixed-line-reservation system was digitized, instantly rendering the analog phones on everyone's table obsolete without offering enough or the right kind of bandwidth for digital devices that might have compensated for the expense of replacing all those phones.  Europe pressed on, but the U.S. deployment of ISDN was stillborn after much hope and hard work going back to the 1960s.  

Large telecoms fail with new technology because it isn't new by the time they finish testing and procurement procedures designed to prevent failure.  As with ISDN, the Bell System invented the DSL modem (Bellcore, 1988), then failed to deploy it before faster "cable modems" from the cable TV industry began to close their window of opportunity.  

The cultural problem of the American telephone industry is an inability to understand blatant shifts in their own industry, 

The future slipped away on slow shifts of complete predictability:

New technologies are flowing out of basic research laboratories in the physical sciences (thank you, Federal government, for the grants) and into products and commercial opportunities (thank you, venture capitalists).  Lobbying, legal challenges and regulatory dilly-dallying are not going to uncork any new technologies. The nation's established telecommunications infrastructure could not make the transition from voice to data.  Its design ordained its failure.  The telecommunications industry's initial response changed from denial (rent a line, let somebody else develop digital transport technology) to failures of imagination: too little, too late. The nation itself had no industrial policy, no forum for choice.  The free market ran its course (creative destruction, boom & bust), 


THE NATURE OF NEW DATA NETWORKS

Packet networks are the alternative to time-division-multiplexed networks, and the Internet has become the ascendant packet network.  Here are some of the major technological differences between them.

PACKETS MUST SINK OR SWIM:  Packets lead a rough-and-tumble existence, dumped into a network where usually no reservations have been made for them.  It may take 1,000 packets to get a long e-mail from Bob to Mary.  These packets may  take different routes, arrive out of order, get lost and never get to their destination.  Nor can any packet mill around forever.  Just as its telomere gives a dividing cell only but so many divisions in life, so an Internet packet's "Time To Live" parameter eventually ends any doomed struggle to reach the destination.  It's tough.  

A packet's salvation is its header.  Each packet is labeled with its origin, its destination, and the service or computer application it supports (e.g., e-mail vs. Web page browsing).  Should a switch mis-direct the packet, the next switch it encounters can read its intended destination and perhaps save the day.  If Bob's packet never arrives, Mary's communications technology can ask that it be re-transmitted.  Knowing they could arrive out of order, Bob's communications technology numbered each packet consecutively as it went out.  Retransmission requests go out as Mary's communications technology tries to reassemble the original e-mail, lines all received packets up in their original order, and sees gaps needing  re-transmission requests.


The chaotic nature of packet transmission of data makes more sense when one recalls that the Department of Defense's Advanced Research Projects Agency (DARPA) issued a request for proposals to build a network able to withstand nuclear attack.  Information would continue to flow from Bob to Mary even if several major American cities were obliterated between them.  The Internet was meant to survive atomic bombing, but a bomb never hit the Internet;  the Internet itself was the bomb that hit the entire nation's telecommunications industry.  

In sum, packets have no circuit reserved for them, no path pre-arranged to take them to their destination.  The packets themselves each say where they need to go.  Packets are like unaccompanied children with their destination address pinned to them, turned over bright and smiling to the airlines system.  

On last century's phone networks, you can tap only one phone line, but on the Internet you can only tap everything because there are no lines.  

The National Security Agency will find no Internet  "circuits" carrying one subscriber's phone calls, no path reserved for e-mails.  Because packets carrying all activities for all users are mixed together, the National Security Agency will have to tap all activities for all users when it is authorized to tap "just"  a phone call or "just" an e-mail sent over the Internet.  

Since the Internet supports all forms of contact with others, which is to say all social activity in personal, civic, professional and commercial life, the Internet offers unprecedented opportunities for surveillance.

ROUTERS DISCOVER ROUTES
: A packet network requires switches that can read the addresses pinned to packets.  Then what?  Instead of commanding other switches to make reservations, each switch was designed to ask any switch within listening distance to identify itself.   Each switch taught itself the topology of the network around itself and made its own decisions about how to route packets once it had peeked inside at the destination address.  Because the switches' key function is making routing decisions, packet switches are called routers.

The first router was an innovative software program running on a general-purpose computer in a university research laboratory.  Specialized hardware followed later when the boy & girl programmers in the San FranCISCO area founded the first – or at least the first commercially successful – router company, Cisco.  

To clinch your grasp of routing, remember that each router has several lines plugged into it.  The request for other routers within listening distance to please identify themselves is broadcast separately on each line, and responses are tabulated separately. The decision of how to route a packet after peeking at its address means deciding which line to ship the packet out again on, given that switches with similar addresses have announced themselves from somewhere down that line but not the others.  

Routing a packet requires looking only at the destination address, which is available early in the header pinned to the packet.  Doing more means looking deeper.  Decoding (demultiplexing)  the Internet to separate from a deluge of packets the individual “applications streams”  (phone, email, Web browsing) requires looking at header items like computer port address numbers.  These identify and distinguish phone application programs  from e-mail or Web-browsing applications.  Distinguishing privileged customers and interesting message senders from others requires looking at the origin address.  Some issues can only be settled by looking beyond the header at the message itself.  

The National Security Agency must push the telecommunications industry to develop and install routers that look beyond every packet's destination address to read who sent the packet, and to sort out what ongoing application flow each packet supports (a long e-mail message, an ongoing telephone-over-Internet call, a visit to a particular Website, etc).   Reading more than the destination address is termed "deep packet inspection".  Machines capable of performing deep packet inspection are only now coming out of labs and onto the market.  Once deep packet inspection has identified a targeted individual and a targeted activity, sending the stream to multiple places for surveillance is trivial.   Until routers performing deep packet inspection become universal, the NSA has had to install specialized equipment from Narus in the rooms where it taps the Internet.   Narus asserts that their "Lawful Intercept module is compliant with CALEA [the Communications Assistance for Law Enforcement Act]. . .[and]  enables packet-level, flow-level, and application-level usage information [to be] captured and analyzed as well as [capturing the]  raw ...  packets [themselves],  for forensic analysis, surveillance or in satisfying regulatory compliance for lawful intercept."
   
PROTOCOLS DEFINE WHAT TO DO WITH PACKETS.  The rules which govern what information has to go into a header and how a header has to be laid out, bit by bit and byte by byte, is called a communications "protocol".  The headers we are discussing are the "IP" headers governed by the "Internet Protocol".   The “IP” protocol is a low-level protocol that is concerned with moving bits through routed networks, just as the “Ethernet protocol” family is concerned with moving bits through the shared Local Area Networks found in offices and, increasingly, homes.   Higher level protocols (Simple Mail Transfer Protocol, Trivial File Transfer Protocol, or Transmission Control Protocol, the “TCP” in “TCP/IP”) might assume the bits and bytes were OK, and specify what to do when a packet failed to arrive within time X.

WHY NOT HELP SOMEONE ELSE WITH **HER** PACKETS?  The devices we connect to our networks – cellphones, PCs – are powerful enough to perform their own routing, yet the country adheres to a hub-and-spoke architecture for all its communications infrastructures.  The free-wheeling, high-growth and rapid-innovation alternative is termed a “fully-meshed, viral network.”

Today we attach each customer separately and make them pay separately.  But customers could act collectively.  By helping one another, users create their own network.  Every laptop computer has two-way wireless connectivity built in ("Wi-Fi).  The self-discovery power of packet networks means that, by running routing software on laptops or cellphones, laptops or cellphones will quickly learn how to pass other people's packets onward.  Recall that is is how Cisco started -- Cisco Systems Inc. was founded on success with a general-purpose computer programmed to act like a multi-protocol router.  

Laptops-cum-routers or cellphones-cum-routers create a self-organizing, self-extensible "mesh" network to serve other users.  WAN (Internet) access, "seeded" to some users, can quickly spread to many.  Greater concentrations of users demanding service create even greater supplies of connectivity to serve that demand, as the pool of machines supporting the mesh rises.  More brings more.  Mesh networks are”viral" since any laptop drawn to it helps enlarge the network.  A viral network is self-extensible without the injection of capital or infrastructure other than by users themselves. 

The emergence of viral mesh networks  http://en.fon.com/   http://meraki.net/  has added impetus to cellphone and telephone companies' moves to limit cities' abilities to create municipal Wi-Fi networks for their citizens.  The mesh would extend the municipality's scattered telephone-pole transmitters and enhance, rather than overload, service wherever people gather.  Although empowered users have most often created applications  (eBay), often in their college dorm rooms (Yahoo, Napster,  Google), mesh networks are an example of empowered users creating -- or at least extending -- the network itself.  

The nation is not always well-served by the response of its corporations to change.  Cellphone companies’ restrictions on the bandwidth, functionality and devices which can be used on their networks has been linked to the
United State’s failure to lead the world’s cellphone industry (Tim Wu, “Wireless Net Neutrality: Cellular Carterfone on Mobile Networks”, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=962027).  Google has lobbied to lift the restrictions on bandwidth, functionality and devices which can be connected to any wireless networks set up in the 700 MHz bands slated for auction in January, 2008.  By permitting different devices from different manufacturers to use the network, or permitting new software to be loaded onto existing devices, Google opens the door to virally (contagiously) extensible networks acting collectively to displace legacy cellphone architectures.  This is not a fight between Company "A" vs. Company "B".  It is a fight over the country's choice to sit still or regain world leadership.  

Next, why can changes which would crash the national phone system be done so often and so successfully on the Internet?    

BANDWIDTH COMES FROM STATISTICAL MULTIPLEXING:    With packets streaming through routers, where is the bandwidth?  

Bandwidth can be delivered because there are no fixed reservations.  At the edge of the network where customers line up for access to services, many will be connected to a single router, and the router in turn connects to network trunks with greater capacity than any customer's PC demands.  If a customer is impatient to send a photo to a friend, the router can pass his packets out as fast as he can deliver them.  Traffic jams are brief because the network is packetized.  Packets can wait in a buffer or leave with incredible speed, to exploit every available opportunity for transport which comes up in the network.  There is no fixed "eight-thousandths-of-a-second" rule as with telephony's TDM (multiplexing).

 
At the network core, after all the customers lined up in many different geographical locations have all been served and all their traffic has been combined, there is only a steady drone.  The combined traffic of millions of customers drones through large routers at the network core much as in any other network's core.

Multiplexing is any service provider's road to revenue growth, but this  irregular or "statistical" multiplexing is new.  The bandwidth advantages of statistical multiplexing are greatest at the network's edge, where the United States has little new infrastructure to take advantage of it.

The National Security Agency must initially place routers performing "deep packet inspection" in every neighborhood Central Office or, in the case of cable-TV companies, in the "head end" location.  In our communities at the edge of the network, speeds are slow enough to inspect every packet lined up in a buffer before it is shipped out. As technology improves, surveillance can move away from insecure neighborhood locations to more secret central data exchanges.  

Example:
Edge routers aggregate subscribers onto an OC-12 line running at 622 Megabits/sec.  This line (and hence the router)  takes 1/622 millionths of a second to get out one bit by definition, and 3.2 millionths of a second to get out a packet that is 2000 bits long (a plausible length).  We all have PCs running at a Gigahertz or better, so we can run a program for  3,200 clock pulses as we try to inspect the packet before it must ship out.  Two hundred lines of assembly code (at 16 clock cycles/instruction) is enough to inspect sender and receiver addresses.  Modern pipelining architectures bring us close to 1600 instructions/sec, but in practice, specialized hardware, not general-purpose PCs,  is used for packet inspection.

Inspections deep enough to begin to look at the contents itself (what you said in e-mails) is possible at line speeds today, provided the machines are installed at the slower network edge (2.5 or 10 GHz), not yet the core (40 GHz).  An example is the Biscayne chip from Bay Microsystems, "a programmable classification processor that can parse, classify and police packets and cells at rates up to 16 Gbps, at minimum packet size, regardless of traffic patterns."  An inspection performed “at line speed” delays nothing and no one using the network will be aware of it. 


NEW NETWORKS FOR THE NEW DATA TRAFFIC:  Where are the country's great and not so great networks to carry this traffic?  After long sanctioning construction of only a single network with a national footprint by only a single player (AT&T), the nation came to have four networks by 1990.  They were AT&T, MCI, Sprint, and Williams, an energy company with the bright idea of blowing fiber optic strands down old pipelines for natural gas, producing an 11,000 mile fiber network, later increased to 26,000 route miles and today perhaps the world's only terrestrial network that almost never gets cut by a backhoe.  (Most of Williams' original network was acquired by WorldCom/MCI and passed ultimately (2006) to Verizon.  By then, however, Williams had duplicated it with better, more modern fiber.)  

These are the nation's more limited networks.  For example, as data and other new traffic arrived, Sprint's network lacked the capacity the company needed by a factor of four, a factor of 16, a factor of 32.  Multiplexing was maxed out -- you can't line up any more slots for any more subscribers, because no electronics can squirt out what is already there any faster than it already is.  With little juice left to squeeze from their electronics, Sprint was saved by advances in photonics, which permitted first four, then 16, then 32 different channels to be crammed onto a single maxed-out fiber.  This is DWDM technology, "Dense Wave-Division Multiplexing", a name derived from the different wavelengths of the 32 discriminably different lasers all shining into a single fiber.

The fiber possessed by these early players is already obsolescent.  With the electronics maxed out a decade ago, they now find their photonics is maxing out as well.  New  players can exploit newer photonics and avoid costly electronics more effectively.  

Fiber technology changes.  The most valuable asset to own is not the fiber but the hole in the middle of an empty plastic conduit where new fiber technology can be blown through cheaply and faster than any "dig first" competitor.  Unlike passive copper wires, fibers exhibit large interactions between electromagnetic radiation (the light) and solid matter (the glass).  In fiber optics, the glass is "doped" with controlled impurities, as transistors are, producing ever-new ways to exploit these interactions.  In new photonic crystal fibers, we are learning to manipulate the quantum physics of the light-glass interactions with  nanotechnology.  Most of us think of fibers as wires.  But fibers are an active component in a system that  exploits solid-state physics in ever-emerging ways.  While stockholders may think that once all their fiber is laid, the company in which they have invested is "done", others might say it's done for.  The company best prepared to benefit from continuous advances in optical fiber physics is Level(3), under the leadership of Jim Crowe.  

Meanwhile, the 22 local Bell operating companies that consolidated first into seven Regional  Bell Operating Companies (RBOCs) have now consolidated essentially back into the original AT&T plus Verizon.  These players had no national network to begin with.  They were the nation's local phone companies.   The cable TV industry was born as CATV, "Community Antenna Television," in which Mom and Pop put an antenna up on the mountain and brought the signal down into homes in the valley.  CATV operations consolidated into MSOs, "Metropolitan Systems Operators" who now covered entire metropolitan areas.  But these players possessed no national network either. They are access-at-the-edge players running legacy technology.  

And so it comes to pass that the nation waits for a modern national telecommunications infrastructure on a duopoly that has never built one.



Broadband penetration, per capita. ITU, 2004.     
Data compiled by  International Telecommunications  Union (ITU), Geneva, Switzerland
See also Consumers Union & Consumer Federation of America at
http://www.freepress.net/docs/bbrc2-final.pdf
The decline of the USA will accelerate, as other countries
already have technologies in place offering more bandwidth at lower cost
than our legacy duopolists, the telco and CATVcorporations.


SUCCESS WITH  BURSTY DATA -- NOW WHAT?

We saw how, with everyone's needs "chunked" into packets, it is easy to share the network among many users, one chunk at the time, without hogging.  Bandwidth is flexible -- no router was ever forced into a rigid round-robin of servicing 32,256 slots even if most were silent (no one talking).  If few people are lined up for service, the router can pass a whole string of your packets out as fast as they come in -- a welcome burst of good service, and a far cry from waiting an immutably long time to get your 8 bits out.  Statistical multiplexing handles the bursty PC data that was the downfall of legacy voice networks.  Beneath these technical advantages lies a lot of complexity that few understand and most insiders do not want to acknowledge.  We will do better.  We will acknowledge the Internet's complexity and then figure out why the Internet has been so successful anyway, so explosively innovative despite its complexity.  

EFFICIENCY IS NOT BETTER EVEN THOUGH EVERYONE SAYS SO:  Packet-based data networks are not more efficient than legacy voice networks in the sense of being able to run their trunks closer to full capacity by selling reserved slots to all the space in advance. People speak of packet networks’ “efficiency”, but in fact packet networks like the Internet are designed to be less efficient in their use of cable capacity.  The truth is that the capacities of new-technology networks have grown so fast that old utilization rules are not high on everyone's business plan anymore.  Serving enough users and purposes so that traffic evens out is called "statistical multiplexing". If your service is global, even the day/night cycle evens out.  

Statistical multiplexing is favored by those with new, high-capacity networks while those with more limited legacy networks favor schemes for controlling access to networks, favor limiting the permitted uses of networks, favor charging fees to limit traffic growth instead of expanding the use of transport services, and favor schemes to "improve"  the Internet's packet-routing technology by adding the centralized control and complexity that is such a notable feature of our legacy voice infrastructure.

SIMPLICITY IS NOT BETTER EVEN THOUGH EVERYONE SAYS SO:  Cisco's "Internet Operating System" used by operators to configure and control Cisco routers is said to have 15 million lines of code.  The firmware and software embedded in a core router adds up to about 8 million lines of code.  That's a lot.  In the legacy voice network, the comparable 5ESS switch has 10 to 18 million lines of code. This switch  bears the brunt of setting up (reserving) calls and providing customer billing and services.  In the core, where traffic aggregation has already long since been done, straightforward transport is the order the day, and transport  voice switches run on only about 3 million lines of code.  Simplicity of one box vs another is not the key to understanding where the Internet's success came from.  

So far, the Internet has prospered despite growing complexity of edge and core routers.  Any FCC or Congressional mandate for routers to support “fast lane” customers or to support homeland surveillance will make them more complicated.

In sum, the Internet has roared to greater success than any previous network without either greater simplicity of the computer code used to direct traffic, or greater ability to get any pipe 100% filled with traffic ("efficiency").  The actual  technical advantages of the Internet are two:  

1. BURSTY DATA IS HANDLED WELL:  The best parts of the nation's telecommunications infrastructure are all-digital, not analog; they increasingly integrate cheap photonics and expel expensive electronics; and they are designed for data not voice because they are packetized, not Time-Division Multiplexed.  A packetized data infrastructure  (Internet Protocol or "IP" packets, Ethernet packets) can handle the sudden bursts in demand for transport -- and the long silences afterwards -- that distinguish data from voice traffic. Internet transport is built on this packetized infrastructure; the Internet's ability to handle bursty data is its key technical advantage.  If no one else is waiting, the user can get an entire fiber (an entire router port) leading out of the central office, not a slice of it, and finish her task quickly.

2. USERS DO NOT NEED PHONE BOOKS:  It is also pleasant to have free "411" service built into the Internet in the form of the Domain Name Service (DNS). Both contacting a Websites (to have a look at it) and sending e-mail require you to go to a particular computer with a particular numerical address.  These machines are the  Web serving computer that holds the pages you wish to see, or to the e-mail serving computer that holds the e-mail account of the person you wish to reach.  Yet we all address each other by our e-mail names (not numbers), and no one has to type in 64.233.167.99 -- oops, changed to 74.125.228.1) in order to get a Web browser to go to "google.com"  (although if you do, it will).  Before a new Web page comes up on your screen, your PC has already asked for and received the needed address number from a Domain Name Server with a "telephone book" of human-meaningful domain names.  

SOMETHING ELSE IS GOING ON.  WHAT IS IT?  But this is not enough.  The success of the Internet cannot be deduced from technology as certainly as the failure of our voice infrastructure was pre-ordained by it  -- the eclipse and sunset of  X.25 packet networks, Fidonet, Bulletin Board Systems, Frame Relay networks, Asynchronous Transfer Mode mini-packet networks, Integrated Services Digital Networks, the Bell System itself, wires, switches and all.  You, poor reader, have endured a long Mini-Tutorial for seemingly little reward, since its underlying telecom technology no longer drives the Internet's future.  Because technology no longer drives the Internet's success, technology cannot prevent the Internet's failure.  And that, dear reader, is why we are here.  We are here to pass the baton.  Success of the Internet Age, born with the new century, depends on Internet philosophies (of design), cultures (of large social communities developing/maintaining technologies) and political empowerments (of users creating new things for networks to do).  

We end with a look at two changes in what the Internet could do (or what users could suddenly do with the Internet) that were stunning in their unimagined novelty and breathtaking in their swift ascent.  Why does this keep happening on the Internet but yet it never happened once on the nation's prior, ascendant national infrastructure, the phone system?  Why does it keep happening?  The Internet's future depends on the willingness of society to find and to accept the answer to this question.  
 
TOTALLY NEW APPLICATIONS RIDE THE NEW NETWORKS

The novel changes in Internet technology that we will take as case studies are Napster and BitTorrent.  

Napster logo (tm).THE ORIGINAL NAPSTER:  After graduating from Harwich High School in 1998, Shawn Fanning enrolled at Northeastern University. His freshman roommate complained that there was no easy way to find MP3s on the Net. Search engines were often out of date, leaving dead links where the music had been. Shawn surmised there could  be a way to combine the breadth of search engines with the "presence awareness" of systems like instant messaging (IM), that  tell you who on your "buddy list"  is signed on at any given time.  The link would not be dead if you knew the person was there now.  The new system wouldn't incur high bandwidth costs.  By leaving all users to store their music on their own computers, the  new system could  just connect  a pair of people and then drop its own connection and go away.   Shawn solicited his friends for help  (April-May 1999), including Sean Parker (also 19 at the time; met him online), Evan Brewer (ran the first server holding the new application) and Jordan Ritter (took over development of the server half of the software).   Three people did most of the work of creating the new technology.  

The new system was put up  in the Spring of 1999.  It had 25 million registered users and a lawsuit by the Recording Industry Association of American (RIAA) in less than a near.  On Sunday, 11 February 2000, I observed 1,768,458 files available for download, representing 7.5 terabytes of data.  Out of bits and pieces of hard drives on 8,981 PCs around the world, the Internet had synthesized a 7.5 terabyte distributed storage system--a capacity that would have  cost  $5 million to buy as a big cabinet less than 10 years earlier.  

As a matter of policy, Napster only reported the portion of their system being handled by whatever particular server a user like myself had landed on that day in February.  The total  size of the online community frequently reached one million concurrent users (November 2000), not the 8,981 shared libraries I saw.  Multiply my numbers by 100.  


BITTORRENT:  One individual conceived, developed and launched BitTorrent, making fast file transfer on slow networks possible.  Files transferred faster than before can be bigger than before.  BitTorrent users began exchanging movies in a nation still mailing red Netflix envelopes.  

BitTorrent logoWhat was new?  When Bob sends a long e-mail to Mary, Internet IP rules or "protocols" govern cutting the mail up into packets, and the Internet TCP rules or protocol dictate how software anyone writes is supposed to count the packets and do something if one of them never makes it.  Why stop at Bob and Mary?  Bram Cohen, BitTorrent's inventor, created software that broke any long file into jumbo-sized packets that he controlled.  Many Internet Protocol (IP) packets are encapsulated inside one Bram Cohen BitTorrent packet.  The Internet's TCP/IP protocols take care of delivering everything in one jumbo-packet.  Bram Cohen's BitTorrent software keeps track of all the jumbo packets of a given file.  The jumbo packets come to be owned by users all around the world.  If you want a particular file, dozens of people around the world can contribute simultaneously to getting it to you.  When delivery streaming in on a globally-based torrent is complete, Bram Cohen's software checks the final file's integrity. It's not Bob and Mary's email anymore.  

Cohen added a social insight to this technical innovation.  A movie might be 1,000 jumbo packets long.  Cohen's social insight was to require every user to upload his packets to other users even if she only had one of them, as a pre-condition to downloading any of the thousand additional packets she might still need and be impatient to receive.   The specification for BitTorrent -- what the sources (hosts) have to do, what clients do to start receiving, the checks and messaging -- was placed in the public domain.  Every different computer needs a different version of this complex software (Apple, Linux, Windows), different users prefer different clients (small and simple, complex and capable), and all clients on all operating systems will have to be taken through bug fixes and their own evolutionary growth.  Given technology made public to begin with, the user community did all of these things.  

The program had explosive success.  The resulting traffic increase was detectable all across the Internet,  from the access ramps of university campuses and ISPs at the network edge ( http://www.wired.com/techbiz/media/news/2005/04/67202 ), to the network core (harder to measure). 

Today, BitTorrent has cut a deal with Warner Bros. to distribute their movies.

Look at the interplay of technologies here.  You and I each download a BitTorrent program (the "client")  to make our own non-standard  jumbo packets that are pretty long because we're moving files that are pretty long.  Our BitTorrent client insists that we share a jumbo packet, so we begin sending out the many conventional IP packets which the jumbo packet embraces.  Our computer's conventional software  follows internationally agreed upon rules -- not Bram Cohen's -- for making IP packets and using TCP procedures to throttle traffic,  detect no-shows and request re-transmissions if necessary.  It is immaterial that TCP/IP will only deal with one pair of computers at a time, that it breaks each of our jumbo packets up because it prefers chunks 21 times smaller, or that it adds (and strips away again at the destination) all kinds of origin,  addressing and packet information even though our BitTorrent software has already added our own destination and packet number information to each jumbo packets.  

The key lesson here is "layer independence" -- you can do anything you want in your part of the Internet as long as you don't change how you look to others.  BitTorrent's request is no different from Bob and Mary's.  Bob and Mary's little IP packets get strung together into one e-mail and they are happy.  We use the same little IP packets, but, from them, our technology creates and manages jumbo-sized packets that become movies, as BitTorrent continues to scour the Internet, looking for our "wares" and trying to strike more "deals".  

The battle cry is  just deliver the bits.  It is no matter that at the other end a program will convert one stream into an email and another program will convert the other stream into a movie, music library, or major software release.  It is no matter than one program, like the telegraph before it, like the phone system before it, is a message between two users, while the other's functional architecture is a revolutionary many-to-one traffic flow.  The TCP/IP service provider companies will perform the delivery according to TCP/IP rules regardless of whether their transport providers in turn use fiber optic or microwave radio links as the physical signals.  Bram Cohen does not care if he has constructed packets only to have the TCP/IP guys chop them up again their way, as long as his stuff is put back together again at the other end.  Just deliver the bits.   Similarly, the TCP/IP guys do not care if the fiber and laser guys in the layer below them are rebels with a radically new photonics technology and so they are not running the Dense Wave Division Multiplexing on the standard wavelengths officially sanctioned by the International Telecommunications Union.  Just deliver the bits.  Leave the layers independent.  


SUCCESS AT INNOVATION is the GREATEST SUCCESS OF ALL

In terms of millions of lines of code in switches or routers, the Internet is no simpler than the legacy voice infrastructure. Yet how is it that no outsider has ever made a contribution to the nation's voice infrastructure, while 19 year olds are walking out of their dorm rooms, signing up users by the millions, and changing the entertainment industry forever?  

The innovation of people like Tim Berners-Lee (invented the World Wide Web), Shawn Fanning (Napster) and Bram Cohen (BitTorrent) does have its parallel in the voice world. In the 1980s, PCs appeared on thousands of desks, and were joined in hundreds of departmental LANs and drove corporations to seek data transport to scattered branch offices.  In the 1980s, the voice-services Bell System essentially walked away from the data business.
The Bells sold private lines from point "A" to "B" and left it to the corporations to figure out what to connect to them.  The United States became the fountainhead of digital innovation as hubs (stackable, switching, remotely manageable) and ever-faster routers were invented and improved.  Startups swarmed to the Internet, reached billions in market capitalization on sales of real products available nowhere else in the world, and made the NASDAQ the “technology index” it is today.  The innovation took place because the innovators did not have to learn the millions of lines of code in the difficult and risky-to-change SS7 signaling language and ESS5 switches. They were outside the system.  

Internet innovators are in a similar but far superior position to corporations 25 years ago. Someone creating the World Wide Web, Napster or BitTorrent does not have to know the millions of lines of code that make modern packet routers work, nor guarantee the integrity of the Domain Name Service (DNS, the Internet's "411" service for looking up numbers), let alone ask for favors from the fibers and photonics crowd.  So, yes, there is that point of similarity between U.S. corporations forced to invent data networking when the giants of voice networking  turned their backs on them.  They, too, had to do things that had never been done before, invent equipment, create the tools to build a different world.  But there is one overwhelming difference.  That difference has made the Internet a globe-turning event, and left the legacy voice infrastructure unable to meet the nation's needs.  

Unlike the corporations of the 1980s forced to invent data networking by an indifferent Bell System, today's Internet entrepreneur holds an Internet connection to the world in his hands.  Instant global markets of millions have turned twenty-somethings into multi-millionaires before capital investments in factories, before human resources and ID badges, before a global sales force of feet on the street, before they are thirty.   Is this the Yankee can-do spirit at its finest, or what?


SUMMING UP SUCCESS

1. LAYER INDEPENDENCE
"Layer independence" summarizes the idea of being able to create something new under the sun without having to master other parts of the system, and without having to get permission from other people running other parts of the system.  

The opposite of layer independence is a transport operator claiming the right to refuse service to some applications, to refuse connection to some devices (the Deutsche Bundespost and modems) or claiming the right to make some applications run slower than others after everyone has paid his access fees and climbed aboard.  "Just transport the bits" reply critics, who say data transport, like rail transport in another century, is a national infrastructure that should be treated as a common carrier with publicly posted tariffs.  If you pay the tariff, you can ride the system.  Anything else stifles the nation's rush to a brighter future.  Just deliver the bits.  

Application inventors sit "on top of the stack", with no one in a higher layer assuming everything below will work.  But some  entrepreneurs make changes to deeper layers.  They, too, depend on those below them to handle their traffic as long as they pay for it, but now obligations to those above arise as well.  The startup  Fonera, offering software to convert any laptop into a router, is creating a TCP/IP network on a physical medium (wireless) and in geographical locations where one never existed before.  It's their router and it does not have to master or mimic the 8 million lines of code in a big Cisco core router, but it certainly must conform to the TCP/IP rules for handling the packets of others (other applications like Bob and Mary's email).  Conforming to rules is easier when the rules are public within all of civic society and available on the open Internet, rather than deemed proprietary and thus sequestered as intellectual property under corporate ownership. The Internet in its early successful growth phase was just such an open community.  

2. OPEN COMMUNITY, OPEN SOURCE
The nation's data infrastructure has been a stunning success compared to the lack of innovation in the older voice infrastructure because all standards are public, and much software is provided to civic society as open source technology.  Who supports this technology?

The legacy voice network employed about 1 million people at its peak to keep it going.  Currently, Telcordia (formerly Bellcore) employs 3,000 professionals to maintain over 100 million lines of code and 150 billing, provisioning and other support systems for the nation's voice infrastructure.  The Internet is supported by volunteers self-selected and able to survive as peers in a community of experts.  The Bell System had one million employees.  The Internet is supported by volunteers.  I think we need to look at this more closely.  

The Internet has a process for proposing drafts and bringing them to RFCs (Request for Comments).  Servers run mailing lists and you (Seriously!  Anyone.) can subscribe to RFC lists and discuss arcana with wonks.  RFCs, when done, are in effect standards for the Internet.  RFCs are  passed (become network standards)  or rejected by rough consensus by everyone who joined the open mailing list of experts.   Internet technology creates Internet technology:  Internet technology like the mailing list is used to support collaboration and create a virtual community of experts who sometimes even meet face to face under the auspices of the Internet Engineering Task Force (IETF)   https://datatracker.ietf.org/public/meeting_materials.cgi?meeting_num=66

Organizations like the IETF and the World Wide Web Consortium are more responsive to public needs and more inclusive of civic input than formal industrial standards organizations like the  International Telecommunications Union ITU-T (formerly CCITT, Comité consultatif international téléphonique et télégraphique), and have done well in guiding rapidly changing technologies that have a big impact on private and commercial life. To change the world, consult it first.

MIT professor Dave Clark, a graduate student of Jerry Saltzer at MIT and now himself one of the grand guys of the Internet, may have unintentionally written the IETF anthem in his A Cloudy Crystal Ball/Apocalypse Now presentation at the 24th annual July 1992 IETF conference. Today, it's immortalized on T-shirts: "We reject: kings, presidents, and voting. We believe in: rough consensus and running code." Which might translate to, "In the IETF, we don't allow caucusing, lobbying, and charismatic leaders to chart our path, but when something out on the Net really seems to work and makes sense to most of us, that's the path we'll adopt."  
http://wired.com/wired/archive/3.10/ietf_pr.html  If only they knew more technology, this is the heaven that every Congressman would want to enter.  


Rough consensus and running code.  
Japanese students study the difference between the unique and uniquely American
Internet Engineering Task Force
and more conventional  
industrial standards organizations  such as the
International Telecommunications Union and the  International Standards Organization


Single individuals creating a new Internet application (e.g., BitTorrent) pursue the same open approach as the Internet itself.  By spelling out how his application works ( http://wiki.theory.org/BitTorrentSpecification ), many third parties are encouraged and enabled to support the innovator's new initiative.   Others can create applications that are "aware of" and use his new way of using the Internet, and Internet technology is extended to many countries and millions of lives.  

3. USER EMPOWERMENT
After 38 years the 911 emergency calling system is still not fully implemented but innovation continues to accelerate on the Internet.  There,  individuals have had a global impact.  World-beating innovation comes from politically empowered users.  An  Internet user who wants to try something new does not have to ask permission.  

One network
stifled American inventiveness, the other propelled it and us beyond anything the world had seen before.  For what kind of person is this a hard choice?  

Success at innovation has been the greatest success of all.  
The idea of building a single-purposed national infrastructure  over and over, one for telephones, one for television, one for Internet browsing and email, another for movie viewing -- all that is archaic. We are innovating our way towards a single national telecommunications infrastructure that is a platform for endless innovation, not a way of serving needs passed and increasingly forgotten,  not just for "seeing what's on TV".   The future holds no "telephone industry."  The future holds no "cable TV industry".  


GRADUATION  MESSAGE

Thank you for completing the Mini-Tutorial, doubly so if you made it despite scant love of tedious technology.

This backgrounder should convince anyone now striving to extend constitutional law and public policy
(the "civic engineers")  to the nation's new digital infrastructures that the creativity demanded of civic engineers today probably exceeds the demands on the people who created the networks in the first place.   We technologists of the information revolution stand on the shoulders of those who made revolutions in chips and computer electronics, in photonics and communications networks.  We hand you a revolution of such impact on the daily lives of social creatures that it is now a social revolution, not a technical one.  As a social revolution, it is up to people who understand civic policy and political culture to guide it.  

Technology was destiny for the declining voice providers.  The technology of Time Division Multiplexing set the industry's fate long in advance, even though there was no national recognition of the obvious, even though there was no national plan for a replacement infrastructure, no national industrial policy in a nation used to getting everything and making few choices.  The nation's doomed central office switches are worth about $250 billion (some say $500 billion).  Perhaps some of the rest can still be used.  Man can never read the writing on the wall until his back is up against it.  

Technology cannot guarantee the success of the packet-based Internet even if technology did forecast the failure of the nation's voice infrastructure.  

The Internet's success comes from
  1. a philosophy of independence between operating layers, 
  2. a culture of openness -- not copyright, not corporate ownership, but openness for standards & specifications ("protocols") and even source code, and from
  3. political empowerment which lets users create new applications to run on the applications layer of the network without having to get permission from service providers who operate other layers of the network.  
Of course, if independence of layers is maintained, the only thing that can happen to operators of other (transport) layers from an explosion of innovation and creativity in the user community is greater revenues and earnings -- new applications attract more business, create new markets, and enable people to live, play and work in ways that used to be impossible (but which now  require buying more network transport service ! ).  What kind of person is uncomfortable with such a future?

Because the success of the Internet comes less from the use of a particular technology ("packet", "router") and more from a broad philosophy of design, from a culture of openness, and from political empowerment of users, a technologist like me cannot promise you the Internet's continued success.  

And what a technological success it is.  A technology that enriches entertainment, human contact, and everyone's access to the cultural triumphs of their civilization can renew civic society and American geopolitical power in the 21st century.  But the Internet success story  is also a technology that can only be sustained by a civic society that understands it.  
The nation can walk away from the culture and values that made the Internet a success.

At some point, the people in a technological civilization must understand their technology or lose their civilization.  

I hope this quick tour of a tumultuous time encourages you to embrace the Internet's philosophy, embrace its culture, and  protect its political empowerment of users.  Empowered users have innovated where the centrally-managed, "command economy" of the voice network stagnated.  

The Internet is an American success story, Department of Defense, National Science Foundation, volunteers and all.  But alas, "
...the Internet and e-mail are the most surveillance-friendly media ever  devised."
--M.A. Caloyannides, Mitretek Systems, writing in IEEE SPECTRUM, May 2000 p. 47.
("IEEE" is a professional society of engineers, the
Institute for Electrical and Electronic Engineers.)

Analog or digital?  Time Division Multiplex switches or packet routers?   Electronic or photonic?  These choices have all been made, but civic society has not yet chosen between surveillance and free innovation.  Surveillance is the dark side of  the Internet.   The risk is that the United States will get the surveillance as the rest of the world gets the innovation.  Others will enjoy the innovation that once made us proud, and others will capture the prosperous growth that once made us the envy of the world.

 


Jerry Nelson, Ph.D. 
Institute for Electrical and Electronic Engineers
Society for Neuroscience
Optical Society of America

jerry-va  removethistext at speakeasy.net
26 Sept2006

FURTHER INFORMATION

This mini-tutorial was written as a backgrounder for those trying to understand the warrantless wiretapping program run by the National Security Agency (NSA) in secret, and in contravention of the Foreign Intelligence Surveillance Act passed by Congress.  How NSA's system works and what it does is easier to fathom than society's inability to respond to it.     

For information on the Foreign Intelligence Surveillance Act itself, the court that administers it, and many related links, see the Federation of American Scientists: 
http://fas.org/irp/agency/doj/fisa/

To look up technical terms, enter requests like these into your search engine:
define teraflops, or
teraflops wiki
supercomputer glossary  or, supercomputer ~glossary

To look up any Representative in the House of Representatives ("The House") or any Senator in the Senate:
http://www.house.gov/
http://senate.gov/ - the "Senators" link lists them.
To find your particular Representative and two Senators:
http://whoismyrepresentative.com/

This is the Mini-Tutorial on telecom networks and the Internet.
 

top      How NSA does it--just the facts       Intro to both papers       home          

Joel Pett, Lexington Herald Leader  2007 - Listen to your old calls at the Bush Presidential Library
Rev 10Nov2011 anchor at Nat'l Internet Rank
Rev 20Jun2012: typos, twisted sentences. here and there.  
    Rewrite summation of Internet's advantages and complexities prior to section on Napster and BitTorrent innovations.
    BitTorrent: distinguish JumboPackets of Torrents from TCP/IP conventional packets. 
Rev21Jun2012: after CDRs, suggest CDR proof of conversations forwarded to govt in realtime is all but certain.
    Fix bad links.