My own experience
This reminded me of a similar incident that occurred when I was an undergraduate in college taking my first non-introductory systems class. We had to write a multi-threaded web server, and I had been banging on the project pretty regularly for at least a week. I had gotten to the point where I was confident my code was stable and ready to be turned in when the program suddenly started segfaulting frequently and without much perturbation. Opening the core file in gdb showed that segfaults were happening when an obviously invalid pointer was being dereferenced, which made no sense. I was reasonably certain that there is no way I could have missed an frequent, obvious segfault problem, so I was a bit perplexed. Then, while I was trying to debug the issue, other applications started to crash, like emacs, gaim and xterm. At that point I was actually relieved that it was probably a hardware issue. I rebooted into memtest86 and thousands of memory errors were detected within minutes. I took out each DIMM and retested and eventually narrowed it down to a single module, and continued using my system with half the RAM until the memory was RMA’d.

NULL with one bit flipped!
So suddenly what I was seeing in the debugger made sense. All of the invalid pointers were powers of 2 and unusual addresses like 0×00000008. After I ran memtest, it clicked — on x86 (and many platforms) a C NULL pointer’s runtime representation is the address with all 0 bits. If one of those bits gets flipped, the standard NULL pointer checks fail and it will probably end up getting dereferenced like a valid pointer shortly thereafter.

ECC and more advanced things
I’m not an expert on DRAM hardware, but it’s possible that the bit error rate is not decreasing as fast as RAM capacity is increasing.* As more and more data is being exclusively kept in RAM (like giant memcached clusters), it is possible bit-errors could becoming increasingly problematic. ECC has generally been considered essential in servers (a paper from Google and University of Toronto in SIGMETRICS ’09 on this is interesting). And I make sure any machines with large storage have ECC RAM (especially ones with software RAID, since so much data manipulation happens in host memory). But now high-end machines are getting much more advanced memory error correction capabilities. For example, IBM’s xSeries servers have features they call “Active Memory”, including memory mirroring (like RAID-1 for RAM), Chipkill — a very strong ECC implementation capable of handling multiple bit errors even from the same memory module, and memory scrubbing. The latter is very neat — basically the RAM equivalent of RAID scrubbing, which is an important process for maintaining disk arrays. Scrubbing detects latent errors before a failure; otherwise they will be detected during rebuild when it may be impossible to recover.

* This problem can affect non-volatile storage, too (the error rate on large media is decreasing slightly slower than storage density is growing; that means single parity redundancy is increasingly likely to encounter an error during rebuild as disk sizes increase. See “Triple-Parity RAID and Beyond” for a bit on that issue as well as the more important throughput to capacity ratio consideration).

The main impetus for my post was not specifically on Panache, however. Over the years I’ve been thinking about the evolution of storage as the scale grows, and it comes up in conversation maybe once every six months. Most recently, it came up during a job interview when talking about some of the projects I’d worked on. I made a note to myself to post something on this.

The growth of storage solutions tends to follow a certain trajectory. In the following I’m going to be very loose with definitions and paint with a broad brush. True backup and redundancy for availability are different, but here I’m just covering redundancy for availability:

• Single drive / JBOD — home users tend to start out with one hard drive or a collection of disks; there’s no redundancy here so they may just manually sync to another drive or system to guard against failure (or the most popular form of failure contingency: nothing).
• Single-host RAID / NAS — after home users’ storage needs increase in scale, they may move to a RAID solution which adds some disk-level redundancy. RAID isn’t a backup solution (you can still delete or corrupt your own files by accident), but it helps give you some redundancy in the face of drive failure. Maybe these drive arrays are in a home-network visible NAS exporting storage via NFS or CIFS or maybe they’re just in one main system where you use the storage locally. Small enterprises may use this kind of NAS solution too. Crude scaling “up” may be achieved by having several different NAS appliances serving separate sets of files.
• SAN / “Enterprise” storage — after you reach a certain level of scale, you want to eliminate the single point of failure imposed by the NAS / single host with a RAID setup. You’re protected from a drive failure, but the host could still fail, the network connection to the host could fail, the drive controller could fail, etc. At this point you tend to move to a storage area network (SAN) setup, often from a company like NetApp, IBM or Panasas. The idea behind a SAN is that your machines connect to your disks like switched local networking. You may have 16 machines and 128 disks, and all machines can talk to any disk. If a machine fails or a controller fails, the rest of the machines can still talk to the same disks just fine. Generally there is some redundancy at the IO controller level, too. These are typically coordinated by a shared-disk filesystem like GPFS or OCFS2.
  In a large enterprise, it is impractical for all machines that need to access the storage to have a direct SAN connection, so in this case you may have a small number of storage-connected NAS hosts that export the same coherent underlying shared-disk filesystem via NFS or CIFS. Then clients can access any exporting host via a standard network filesystem protocol, and any NAS host can die without the entire filesystem becoming inaccessible. Redundancy may be achieved by layering the filesystem over redundant disk arrays in the SAN, or by doing parity / Reed Solomon coding within the filesystem on individual disks (I talked about “Declustered RAID” in an earlier post), or by doing file-level and metadata-level replication.
• Google Filesystem — “Enterprise” storage based on SANs is way more expensive per unit of storage than commodity disks, and there is a limit to how far you can scale “out” the number of hosts with access to the storage and scale “up” the amount of storage. Once you get to the level of companies like Google or Amazon, you are already going to have sophisticated internal software for managing failures when dealing with your own computation. At this level of scale, it makes sense to build your own custom data storage solution using the same failure-handling principles with unshared, commodity disks. Google filesystem (GFS) is layered over a bunch of commodity servers with local non-redundant storage. The storage is aggregated into a coherent distributed filesystem and data is replicated at the “chunk” level (a large sub-unit of a file). This will cost a lot less for more storage and you can build in all kinds of special considerations for your own uses (e.g. MapReduce works in concert with GFS to try to process data locally where it is stored).

As I mentioned above, I’m really glossing over a lot here and speaking in generalities. Some things I’ve glossed over:

• In reality there are very large scale systems that do use SANs (i.e. HPC clusters for scientific simulation) rather than local storage and custom software. Batch, shared-nothing crunching via MapReduce has different data requirements than tightly coupled parallel MPI jobs. The final stage of “scale” listed above is really about what you need to do with all of that data.
• GFS/HDFS aren’t replacements for traditional POSIX filesystems — they are accessed through a library, they don’t provide the same semantics (GFS has atomic appends which may append the data more than once), and they were designed to be used in a specific context for batch data analysis in concert with MapReduce/Hadoop. Given the current state of development, you probably wouldn’t archive important datasets within HDFS. The finer details of Google’s own strategies are private, but given the intended uses of BigTable, it’s probable that a lot of data is actually archived within GFS on top of BigTable. This makes sense if the intended uses of the datasets are all within the framework of MapReduce — jobs that can read out of BigTable.
• Large internet companies with these specialized storage solutions still probably use SAN-based systems for some purposes. It may not make sense, for example, to store source repositories in GFS or BigTable or to try to export data in GFS like an NFS share.
• There are, of course, many other things that don’t fit so neatly into these categories. For example, a while back I wrote a post where I talked about distributed peer-to-peer filesystems that run over wide-area networks (often built on DHTs).

As an aside, it’s interesting how there is an somewhat parallel scaling trajectory to databases as well. You start off with stock relational databases, then you tend to shard and denormalize, then from there you may loosen semantics further, interpose memcached, try to do custom transactional replication, etc.; if that can’t keep up, you may eventually progress to your own eventually consistent key-value store or document DB or non-relational tabular data storage solution or any other number of interesting variants. But I will save that for some future post.

But sometimes there are structural asymmetries in the system that make starvation more likely and fairness more important. For example, reader-writer locks make it easier to starve writers if more readers can always enter while the lock is in reader mode (because writers must wait until all readers are done before acquiring). This brings me to the traffic scenario below:

A popular drive-through (Chick-fil-a) gets backed up at meal times. The driveway in the picture above is the drive-through entrance. Now, once the driveway gets backed up to the road, new cars cannot enter until the line moves forward some. A car turning right into the entrance must wait if the line is backed up too far, but a car turning left across traffic must wait for both 1) no immediate oncoming traffic and 2) space in the line. Since the left turning car must yield to all right turns, you get an interesting starvation phenomenon: the line backs up so no car can enter and the left turning car waits. The line moves forward some so now there is room, but the traffic in the other lane continues to move and blocks the turn. Eventually a right turning car fills the space in the line, so when the next break in traffic occurs the left turning car still cannot complete the turn. All of the right turning cars occupy any freed space in line and starve out the left turning ones.

This was a frustration I encountered first hand, and it reminded me of an article I read a few weeks ago (on Slashfood) about increasingly unreasonable fast food drive-through wait times and the traffic problems a poorly-placed drive-through can cause.

“In Peabody, Mass., the opening of the first New England Sonic in August drew excited customers from across the region anticipating their first extra-long chili-cheese Coney hot dog. In the first month, it wasn’t unusual for customers to wait in their cars for up to four hours.”

Not exactly “fast food,” is it?

Anyway, I decided to see if I could whip up a quick and dirty Chrome extension to substitute for mozex. And when I say quick and dirty, I mean it — this code would make a Perl-happy sysadmin blush. But I hacked something up in a few hours, and I thought I’d put it out there since I find it useful. Maybe someone else can run with it and make it more complete like mozex. With a few one-line edits, this extension can be made to allow the editing of virtually any textarea in any external editor. I limited it to Gmail because I don’t know if there is the possibility of resource leaks or other weird side-effects from running it more extensively. Also, it’s hard-coded to use Emacs, but you could just as easily use Gvim (have it spawn an xterm with an editor or use emacs-client or whatever).

I’ve put the extension source files up here:
http://www.thegibson.org/blog/files/emacs_chrome

Edit: see http://www.thegibson.org/blog/files/emacs_chrome/sync_ver — the original only works right on platforms like Linux due to a serendipitous interaction (see comments).
Edit2: see http://www.thegibson.org/blog/files/emacs_chrome/ba_sync_ver/ — this version uses a Chrome “browser action” rather than a “page action.” The “browser action” extension type is better suited for this. Note that some of the text below is not longer applicable to this new and simpler version, so see comments. I’ll try to make a new post soon with the updated info.

The extension comes with a Python script (you’ll need Python 2.6 to run it). The pycl.py script is a little web server running on port 9292. Run the web server and leave it in the background. Then add the extension to Chrome (go to chrome://extensions). Once you add the extension, you can load up Gmail and either compose or reply to a plain text message (not HTML). Then use the “compose in a new window” icon to open a new window, and you’ll see an Emacs 23 icon appear in the “page action” area (the upper right-hand corner of the address bar, where the SSL lock icon goes). Clicking the icon sends the contents of the text area to the Python web server, which writes it to a temporary file (using tempfile.NamedTemporaryFile) and spawns an editor. Edit your text; when you’re done, save and exit. In the meantime, the background script will start polling the web server to see if the editor has terminated. When the editor finishes, the file contents are read and sent back to Chrome, which modifies the textarea with the new contents.

Anyway, it has several major limitations — first and foremost, it only works consistently if you compose or reply in a new window (click that little diagonal arrow icon in the upper right hand corner of the message frame). Sometimes it works on the main Gmail page, but not always. Due to the hooks it uses, if you open the new window while viewing the message, and then click reply, you’ll have to type something in the textarea or un-focus and re-focus the window before it’ll “notice” the existence of the new textarea. This is a limitation of the way the extension content script is finding the textarea in the page. Maybe the fact that it only works consistently in a new window has to do with Gmail’s crazy dynamic DOM manipulation. Or maybe — perhaps even more likely — I’m missing something obvious. I’m sure someone with more Javascript / DOM / experience could probably fix this. I’ve never used Javascript in any context before, so it was an interesting experience. I basically looked at two Chrome example extensions — 1) the RSS feed subscription and 2) the Gmail checker — and munged their code and techniques together.

Changing editors
To change editors, just edit pycl.py and modify the line:

p = subprocess.Popen(["/usr/bin/emacs", f.name])

Making it work on arbitrary textareas
If you want to change the extension to work on more than just Gmail, first edit majifest.json and change the “matches” line:

"matches": ["http://mail.google.com/*", "https://mail.google.com/*", "file://*/*"],

"matches": ["http://*/*", "https://*/*", "file://*/*"],

The second matches example works on all http and https websites.

Then, in the ta_find.js file, change the getElementsByName(‘body’); line:

  result = document.getElementsByName('body');

  result = document.getElementsByTagName('textarea');

The second will find any textarea on the page (one limitation of the current incarnation is that it will only deal with the first textarea, however).

Making it not suck
This is my first foray into in-browser Javascript programming. I’m sure there’s a lot of missing exception handling in background.html when dealing with the XmlHttpRequest objects. There might be resource leaks too. And there are corner cases where it won’t get the textarea contents to send to the external editor correct (like if you only use mouse cut and paste to change the contents around without any keypresses). Also, in the Python script, there’s not much error handling either, and using a dedicated private directory (like mozex) rather than tempfile.NamedTemporaryFile is probably advisable on shared systems. Also, it’d be great if some Javascript wizard could help figure out why it only works with Gmail when you open a new window. Ideally I would have liked to add a context menu item to all textareas, but I figured it was easier to use existing Chrome extensions as templates and do it with the “page action” mechanism.

Thoughts on Javascript
Looking at the way that the Gmail checker and the RSS feed subscription work, I see an extensive use of closures and continuation-passing style asynchronous programming. Given JavaScript’s reputation, this is definitely a lot nicer and cleaner than than what I expected. Steve Yegge has also pointed out several times that JavaScript is a nicer language than people actually give it credit for, and now I know firsthand.

The paper introduces BPFS, a filesystem for BPRAM which exploits various properties of directly-addressable non-volatile memory to improve performance and durability over traditional filesystems. It’s interesting how a lot of the very thorny problems of filesystem consistency associated with block-based disk interfaces are solved elegantly by applying traditional in-memory style atomic updates to data structures in BPRAM. Traditional filesystems use relatively hairy techniques like soft-updates or journaling to try to ensure that persistent data structures are updated in a way that preserves metadata consistency even if the system fails in the middle of a write. With byte-modifiable data structures, you can use atomic update techniques similar to those used in lock-free data structures (e.g. do modifications on a private copy of data and then atomically “publish” it by setting a pointer field pointing to the private copy — and make sure to put architecture-appropriate fences so reordering doesn’t bite you!*). When I started reading the paper, two classic systems paper quickly came to mind: “Lightweight Recoverable Virtual Memory” and “Free Transactions with Rio Vista”. Rio Vista owes a lot to RVM, but one of the memorable things about Rio Vista is that it uses battery-backed RAM to perform atomic and durable transactions on persistent data structures. Like BPRAM, the persistent but directly-modifiable nature of battery-backed DRAM really simplifies many aspects of the system. Naturally, when I got to the end of the paper, I found that they cited Rio Vista in their related work. Anyway, it’s good to see that one of the co-authors is a former Georgia Tech classmate, Derrick Coetzee.

* One of the most interesting parts of the paper is their “epoch” mechanism — while fences work fine for volatile data structures, they don’t provide strong enough semantics when you have persistent memory. Memory fences aren’t strong enough because they just affect a CPU’s view of memory, not the actual contents of DRAM. A CPU’s view of memory is basically DRAM plus “diffs” of more recent data at various caching levels. With BPRAM, if the power goes out, the diffs in cache disappear and then the persistent data left may not be consistent by itself. You have to make stronger guarantees about when persistent data gets written to maintain proper stored data structure consistency.

Linux and flash
Although the SOSP paper is recent and on my research radar, it is not what prompted this post. The original impetus came from my recent viewing of the Linuxcon 2009 roundtable discussion. This roundtable gained some Slashdot notoriety because Linus made a comment about the kernel being “huge and bloated,” due to its expanding feature set and icache footprint. During the Q&A session, someone asked a question regarding flash RAM. The question was predicated on the assumption that flash will transition to directly addressable (internal) memory and was asking whether, since flash cells have limited lifetimes, would Linux eventually integrate code to deal with directly accessible memory failing. Ted Ts’o sort of dismissed the question and said that he believed that the right place to address failure properties are in the hardware (and he didn’t necessarily agree that flash would move to directly addressable memory). Currently, flash exposed with a “disk drive” interface — i.e., flash that looks like a fast hard drive — handles failure and wear leveling underneath the storage interface.

This reminded me, however, that there is another class of flash support in Linux that is commonly misunderstood. Linux has a MTD (Memory Technology Devices) subsystem which supports “bare flash” devices. These devices are more common on embedded systems, and basically “bare flash” is exposed flash memory that doesn’t look/act like a standard hard drive. The software above gets to access the real flash blocks and has to handle wear leveling, dealing with bad blocks and also dealing with write/erase semantics (things that the firmware would do in a hard drive-like flash disk). MTD devices don’t act like block devices and actually expose three operations: read, write, and erase. There’s a whole class of Linux filesystems built to run on top of these “bare flash” devices: YAFFS, JFFS2, LogFS, and UBIFS, and they’ve also factored out some of the common functionality used in a lot of these filesystems into a separate UBI (Unsorted Block Images) layer. But I see a lot of misunderstanding of the point of these filesystems. Many casual Linux users or observers think that these filesystems are flash-optimized regular filesystems to be used on top of hard-drive like flash devices (or CF/SD/etc.). Anyway, I see this mistake made a lot on forums and such so it came to mind after the Linux roundtable question.

This presentation is updated from “SSH Tips and Tricks given on Wed. Feb 28th, 2007 by Benjamin McMillan and David Hilley.

New things compared to last time:

• ssh -o StrictHostKeyChecking=no
• ssh-keygen -R & HashKnownHosts
• ssh Restricted Shells (rssh, scponly, etc.)
• ssh keychain
• ssh pseudo TTY allocation
• forwarding bind addresses
• PAM & ssh
• rsync & ssh ciphers
• compression
• parallel ssh tools
• fuse sshfs

In this presentation, I will skip the “using ssh” basics.

SSH Config File

The ssh config file isn’t particularly advanced, but many of the later tips benefit from config file customization. Typing your username for every host is a pain. In addition, ssh has a large variety of special options (e.g., compression, agent forwarding, host-specific private keys, etc.) that may differ on a per-host basis. In your ~/.ssh directory, create a file named config to set host aliases and options. Example ~/.ssh/config:

Host feynman
  User foobar
  ForwardAgent yes 

Host hawking 192.168.1.1 router
  Hostname hawking
  User root
  ForwardAgent yes
  Port 222 

Host *.cc.gatech.edu
  User bmcm
  ForwardAgent yes
  Compression yes

Change ssh cipher for rsync

Ever rsync over ssh to CPU-limited devices (e.g., VIA CPUs, ARM, Atom, contended shared servers, etc.)?

• arcfour and blowfish are typically “cheaper” than 3DES or AES; best one depends on CPU/architecture
• “none” is a SSHv1 only option; don’t use with password across a public network — your password will be sent in plain text (keys are okay)
• Note: If you use connection sharing and already have a connection open, the cipher request will be ignored.

SSH Compression

ssh features built-in compression:

SSH Pseudo TTY Allocation

ssh -t forces pseudo-tty allocation. By default, when you ssh without a command to execute (just to log in), a pseudo tty is allocated. When you specify a command to execute, ssh does not allocate a pty. Forcing it is necessary if you want to run something that’s not just plain text output, like screen or top.

SSH Host Keys

When you connect to a new host via ssh, the host key is stored in ~/.ssh/known_hosts, establishing the host’s identity.

StrictHostKeyChecking option

When you first connect to an unknown host via ssh, it asks you to confirm:

HashKnownHosts option

Default in many cases; hashes hostnames in ~/.ssh/known_hosts. Entries look like the following (top is hashing on, bottom is hashing off):

|1|QxEMrKqPTNuBtHIEYSbztjaOkF8=|Y1387YZhibtug9rr4ZVXenyRXb4= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAg...

boar3.almaden.ibm.com,9.1.112.221 ssh-rsa sAAAAB3NzaC1yc2EAAAABIwAAAg...

ssh-keygen -R & -F

When a host’s key changes, ssh will make you remove the offending key from ~/.ssh/known_hosts before it will let you connect (unless you force it to ignore). Some people edit this file by hand (why?), but ssh-keygen has specific options for this:

• ssh-keygen -R host — removes the entry for given host
• ssh-keygen -F host — finds entries for a host

This works whether or not ~/.ssh/known_hosts is hashed, so don’t waste your time editing things by hand.

View a host’s key fingerprint?

Use ssh-keygen: ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub (or dsa). Add -v to get an ASCII art depiction:

$> ssh-keygen -v -l -f /etc/ssh/ssh_host_rsa_key.pub
+--[ RSA 2048]----+
|    oo  ..       |
|   . ..o. .      |
|    . o. +       |
|     . .= .      |
|      .oS+       |
|      ..oo..     |
|         o+ o    |
|          E+ .   |
|           =o    |
+-----------------+

SSH Keys

ssh keys allow you to log in to remote machines with a public/private key pair. In your ~/.ssh directory you will use ssh-keygen to generate a key pair, and you will have to distribute the public half of the key pair to remote machines. You keep the private half of the key pair private (as if that isn’t obvious).

Passphrases

A passphrase is a password that unlocks the generated key. It may be blank, but we suggest only making it blank if you have other security measures (one of those is detailed below). A stolen but password-protected private key will not compromise your accounts.

Generating a key

Distributing the key

Note: Make sure ~/.ssh/authorized_keys, ~/.ssh/ and ~/ do not have go+w perms or sshd will ignore your public keys. “StrictModes no” in sshd_config will disable this check, but not a good idea. Think about it: if ~/.ssh/authorized_keys is writable by another user, they can throw in a newly generated public key and log in as you. If ~/.ssh is writable, they can rename or delete ~/.ssh/authorized_keys and make a new one with their key. If ~/ is writable, they can rename the whole ~/.ssh directory and make a new one to use (and so forth).

Finer-grained Key Control

You can limit the power of keys to run certain commands, to only connect from certain hosts, and disallow things like port forwarding, X forwarding, etc.

Concrete example:
I use a restricted ssh key to run fetchmail. My mailhome is baobab.cc.gatech.edu, and I’ve set up a special entry in my ~/.ssh/config for email checking:

List of things:

• no-agent-forwarding
• no-port-forwarding
• no-pty
• no-X11-forwarding
• command="command"
• environment="NAME=value"
• from="pattern-list"
• permitopen="host:port"
• tunnel="n"

Notes:

• ‘command’ sets a command to run. Adding no-pty makes the command 8-bit clean for two way data transfer. In command, one can use $SSH_ORIGINAL_COMMAND to refer to the command the user wants to execute. Using the shell expression “${SSH_ORIGINAL_COMMAND:-}” handles the case in which no command is specified gracefully [1].
• ‘environment’ modifies the environment variables.
• ‘from’ restricts based on the remote host (see ssh patterns).
• ‘permitopen’ allows -L forwarding for specified host/port combinations only.
• ‘tunnel’ sets up VPN tunnels.

[0] Not sure of the original source of this since I’ve been using it since at least 2003. This might be it, though: http://mah.everybody.org/docs/mail/fetchmail_check
[1] Hint from: http://www.oreilly.com/catalog/sshtdg/chapter/ch11.html



SSH Agent & Single Sign-on

Wouldn’t it be convenient to enter a key’s passphrase only once, and have it unlock your key for the entire terminal session? ssh-agent is a background service that keeps track of your unlocked keys. It can manage multiple keys, and you can add or remove keys or identities dynamically. When you use ssh to login to a server with key authentication, ssh will ask your agent if it already has the key unlocked. That way, you don’t have to type your password every time.

Starting the agent

Sometimes your distribution’s X server will start up the agent automatically. If it doesn’t, you can either start it manually each time, or add it to an environment appropriate startup script. For Gnome, you might put this in /etc/X11/gdm/PreSession/Default.

Adding an identity to the agent

Before things will start working, you must add your key(s) to the agent. You can do this automatically when you login to Gnome by adding it to your Gnome session (Startup Programs). There are several X11-based ssh-agent password prompting programs (ssh-askpass, ksshaskpass, etc.).

Keychain

Keychain is a front-end for ssh-agent allowing easy, system-wide sharing of ssh-agent (rather than per-login). Add something like this to your profile / bash_profile / preferred login script:

PAM SSH

Single sign-on via PAM. Add pam-ssh to XDM’s pam hooks (or GDM or whatever) and your X login will also unlock your private key via ssh-agent.



Port Forwarding

Tunnel traffic through your ssh connection to access non-public / local networks.

Local Forwarding: -L

Say you want to access services local to a private network (not publicly exposed), but from outside you can only ssh to a host on that network. For example, you want to access an internal website running on webhost. You can’t access it from a web browser at home normally, but ssh local port forwarding will let you tunnel the request through the ssh connection to it:

• Ports = 1024. For example: ssh -L 8080:webhost:80 sshhost will bind local port 8080 to tunnel to webhost port 80 through the ssh connection to sshhost. Then you’d just point your browser at http://localhost:8080
• With many websites, virtual hosts are used and the tunneled request won’t have the right virtual host name (i.e., if you point your browser at http://localhost, it’ll use localhost as the virtual hostname rather than webhost). You can fix that by editing /etc/hosts and creating a temporary mapping, but dynamic forwards (see below) are often more convenient.

Here is an example for accessing pop3 via a tunnel. The command below sets up the forwarding and then leaves the connection open in the background without a shell.

• -N tells the client not to execute anything remotely
• -f tells ssh to background

Remote Forwarding: -R

Local forwarding lets you tunnel requests from a local port through the remote host you’ve ssh’d in to. What if you want to go in the other direction — bind a port on the host you’re ssh’d in to and send it back to the host you ssh’d from? Why would you want to do this? Say you have a desktop at work, and there’s no way for you to ssh directly to your work desktop because it doesn’t have a public IP (or incoming ports are firewalled off). ssh from your work desktop back to your home network and use a remote forward. Here is an example ~/.ssh/config:

Dynamic Forwarding (SOCKS Proxy): -D

Use -D to create a SOCKS proxy.

Bind Addresses

All flavors of forwarding — local, remote and dynamic — can optionally take “bind addresses” so you can expose forwarded ports externally. Normally forwarded ports are only listening on loopback interfaces, but there may be a legitimate reason to publicly expose a port forward. Example:

$> ssh -L 2222:localhost:22 localhost  # establish a port 2222 forward to my own port 22
$> ssh -p 2222 `resolveip -s localhost`
The authenticity of host '[127.0.0.1]:2222 ([127.0.0.1]:2222)' can't be established.
...
$> ssh -p 2222 `resolveip -s $HOSTNAME`
ssh: connect to host 143.215.128.82 port 2222: Connection refused

$> ssh -L '*:2222:localhost:22' localhost
$> ssh -p 2222 `resolveip -s $HOSTNAME`
The authenticity of host '[143.215.128.82]:2222 ([143.215.128.82]:2222)' can't be established.

Connection Sharing (ControlMaster)

Use ControlMaster auto to speed up remote filename tab completion, sshfs, or other ssh operations to the same host. New connections to a host will use an already established session (if present), which makes operations a lot faster.

Add to ~/.ssh/config:

Parallel SSH Tools

Good for managing clusters of identical machines (or just all of your own internal systems).

• pssh (parallel-ssh)
• Cluster SSH
• mssh

$> parallel-ssh -l root -h hosts.txt -i -- uptime
[1] 18:45:25 [SUCCESS] capricorn.cc.gt.atl.ga.us 22
 18:45:25 up 1 day, 4:20, 0 users, load average: 0.00, 0.00, 0.00
[2] 18:45:25 [SUCCESS] leo.cc.gt.atl.ga.us 22
 18:45:25 up 1 day, 4:17, 0 users, load average: 0.00, 0.00, 0.00
[3] 18:45:25 [SUCCESS] scorpio.cc.gt.atl.ga.us 22
 18:45:25 up 1 day, 4:13, 0 users, load average: 0.00, 0.00, 0.00
[4] 18:45:25 [SUCCESS] libra.cc.gt.atl.ga.us 22
 18:45:25 up 1 day, 4:12, 0 users, load average: 0.00, 0.00, 0.00

Restricted SSH Shells

Sometimes you want users to be able to ssh to a machine but not perform arbitrary shell functions. A system administrator could rely on restricted keys to do certain things, but restricted shells offer more options.

ChrootDirectory option

Relatively recent addition to OpenSSH:

rssh

Restricted shell for sftp, scp, rsync and cvs. Doesn’t support unison or svn.

scponly

Restricted shell for sftp, scp, rsync, unison and svn. Doesn’t support cvs.



SSH Escape Sequences

Escape sequences are only recognized after a newline and are initiated with a tilde (~) unless you modify it with the -e flag. Hit ~? on a running ssh session to see a list of escapes:

Supported escape sequences:
~. - terminate connection
~B - send a BREAK to the remote system
~C - open a command line
~R - Request rekey (SSH protocol 2 only)
~^Z - suspend ssh
~# - list forwarded connections
~& - background ssh (when waiting for connections to terminate)
~? - this message
~~ - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.) 

VPN Tunneling

Did you know that ssh can do layer 2 and 3 VPN tunneling? Check out ssh -w. Example from manpage:

FUSE SSHfs

Expose a remote host’s files like an NFS mount but via ssh:

I’m a Debian user, so I’m used to having a very large set of packages in the base repositories. On a typical Debian desktop system of mine, I may have the base repos with main, contrib and non-free plus debian-multimedia.org (and backports.org and volatile.debian.org on servers running stable). On RHEL, the set of base packages is much smaller; CentOS (plus the CentOS plus) has a little bit more but basically the same issues. So people tend to add a bunch of other repositories like the EPEL, rpmforge, RPM Fusion, or individual repos included in those umbrellas like Dries, DAG, Livna, etc. With the proliferation of repositories comes duplicated effort and problems with mutual compatibility.

For example, consider loading a RHEL 5.4 system from scratch and adding the EPEL and rpmforge repos:

• Install EPEL: rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm
• Install rpmforge: rpm -Uvh http://packages.sw.be/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.`uname -m`.rpm [0]

On CentOS I would now add yum-priorities (yum install yum-priorities) and set relative priorities for the different repos. But either way, let’s say you now try to install vlc with yum install vlc and it explodes with the following:

vlc-0.9.9a-3.el5.rf.x86_64 from rpmforge has depsolving problems
  --> Missing Dependency: libcucul.so.0()(64bit) is needed by package vlc-0.9.9a-3.el5.rf.x86_64 (rpmforge)
vlc-0.9.9a-3.el5.rf.x86_64 from rpmforge has depsolving problems
  --> Missing Dependency: libdvdread.so.3()(64bit) is needed by package vlc-0.9.9a-3.el5.rf.x86_64 (rpmforge)
Error: Missing Dependency: libdvdread.so.3()(64bit) is needed by package vlc-0.9.9a-3.el5.rf.x86_64 (rpmforge)
Error: Missing Dependency: libcucul.so.0()(64bit) is needed by package vlc-0.9.9a-3.el5.rf.x86_64 (rpmforge)

Why is this happening? Well, for the libcucul.so.0 dependency, the EPEL packages libcaca, and so does rpmforge, but they are not entirely substitutable (there’s a similar problem with libdvdread). Usually I trust the EPEL more (and give it a better priority) since it is more “official” and Redhat-endorsed (and has high quality packaging standards), but in this case we need to satisfy all of the dependencies from rpmforge. So, I use the following command:
yum --disablerepo='epel' install vlc

For the system at the InstallFest, it was even more complex because it had the EPEL, rpmforge and RPM Fusion and some packages from each. I had to disable all but rpmforge and install libcaca and caca-utils and then disable RPM Fusion to properly install vlc (and first I had to remove the libcaca package that was pulled in from the EPEL). yum was developed in part to resolve “rpm dependency hell”, but these warring packaging factions are causing the same problems to be exposed again to the user through yum, which is frustrating.

[0] Right now packages.sw.be seems to be broken in some places so you can grab from say http://mirror.cpsc.ucalgary.ca/mirror/dag/redhat/el5/en/`uname -m`/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.`uname -m`.rpm

So apparently there’s a few different drivers that could support the card, and they all have their quirks. Some people have had success with the open-source and in-kernel megaraid driver, and LSI provides some 1068E Linux drivers on their site. Their drivers are the semi-closed mptsas drivers, but they do provide the source and dkms support, so they can be recompiled against different kernels (provided they aren’t changed to the point where stuff breaks).

Our problem was that the mptsas driver was compiling and loading fine but we weren’t seeing any disks. It turns out the integrated X8DT3 has two modes controlled by a hardware jumper: SR mode (Software RAID Mode) and IT (Integrated Target Mode). SR mode is the default, and that is where the card exports logical arrays to the host OS, but the mptsas driver doesn’t seem to work in that mode. In IT mode, the card just exports individual drives, and the mptsas driver works fine. However, the primary user didn’t want individual drives, he wanted the logically configured arrays (yes, I’m aware that Linux md/software RAID could have probably done just as good a job as the LSI controller’s SR mode, but I wanted to see if I could get it to work in SR mode).

Anyway, a blog post that informed me about the IT/SR mode distinctions had a pointer to Supermicro’s FTP site, and there I found some promising “SR” drivers. I had seen references to a megasr closed source driver in some forum posts, but I couldn’t find it from LSI (at least not a RHEL5u3 version). Well, the megasr is apparently what is on Supermicro’s FTP site. I navigated to ftp://ftp.supermicro.com/driver/SAS/LSI/1064_1068/SR/Driver/Linux/ and grabbed a disk image for the version of RHEL 5 we’re using (luckily we hadn’t updated to the newly released 5.4, because there doesn’t seem to be an image for it). I grabbed the .img file and saw that it was a floppy image using the file command. I mounted it loopback and extracted the compiled megasr.ko using the following process:

• mount -o loop megasr-13.10.0708.2009-1-rhel50-u3-all.img /mnt/
• zcat /mnt/modules.cgz | cpio -idv

The last command will extract the modules to the current directory (making subdirectories for various kernel versions and architectures). The .img file is compiled for use with the RHEL installer, hence the floppy image format and the modules packed as a gzipped cpio archive. We had a working install on a standalone SATA disk and the SAS drives off of the controller were just going to be used for data storage, so we just needed to add the kernel module after the system was already installed.

So I removed the LSI provided modules (with rpm -e to remove the mptlinux dkms package), copied the appropriate megasr.ko to /lib/modules/`uname -r`/extra, ran depmod, rebooted and everything was finally working in SR mode. Luckily we were using RHEL — it looks like people who are using non-RHEL or SLES distros are out of luck because they don’t seem to provide source to recompile the module against vanilla (or other) Linux kernels.

• There are two ways to get to the LVM partition on this disk, and I’ll cover both: 1) the manual offset finding way and 2) the easy way.
• First, the easy way: make sure the loopback module is inserted with the max_part parameter, which causes the automatic creation of loopback subdevices for individual partitions. An easy way to make sure is to remove it and re-insert with the right parameter: modprobe -r loop && modprobe loop max_part=63
• Next, mount the whole disk image loopback: losetup /dev/loop0 sda.img. Now you should see /dev/loop0p1,/dev/loop0p2, etc. for all of the individual partitions. Now you’re already done — you can go directly to the next section to deal with LVM directly.
• If you can’t do the easy method, mount the whole disk image loopback to look at the partition offsets: losetup /dev/loop0 sda.img
• Now that /dev/loop0 looks just like the block device image, so check out the partition table sector offsets with fdisk: fdisk -u -l /dev/loop0. I use sector offsets (-u flag) rather than cylinders because they are easier to work with and some partitions may not fall on cylinder boundaries. My image shows something like this:

Disk /dev/loop0: 250 GB, 250056737280 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488392065 sectors
Units = sectors of 1 * 512 = 512 bytes

     Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1   *          63      401624      200781   83  Linux
/dev/loop0p2          401625   488392064   243987187   8e  Linux LVM

• Now, remove the whole disk image from /dev/loop0: losetup -d /dev/loop0 and set /dev/loop0 to just the LVM partition by adding the partition offset. Here, the second partition starts at sector 401625, and each sector is 512 bytes, so the offset is 205632000. Run losetup /dev/loop0 sda.img -o205632000

Now whichever method you used, you have a loopback device with the LVM physical volume partition: /dev/loop0p? (2 in my case) if you used the easy way, or /dev/loop0 if you used the manual offset method. Get LVM to recognize the physical volume and activate the volume groups:

• Tell LVM to scan for new physical volumes: lvm pvscan
• Activate the volume groups: lvm vgchange -ay (it will print something like 2 logical volume(s) in volume group "VolGroup00" now active).
• Now you can finally mount the LVM logical volumes. Run lvm lvs to list the logical volumes. Each should appear in /dev/mapper, typically with the device name (volume group name)-(logical volume name), like VolGroup00-LogVol00.
• When you are done, unmount all logical volumes and deactivate the volume groups with lvm vgchange -an. Now you can reclaim the loopback device by using losetup -d /dev/loop0.

Garbage Collection
One topic where I didn’t fully agree with Bjarne was the issue of garbage collection. Personally, I think the addition of hooks to support garbage collection to C++0x is an essential decision for enabling effective concurrency programming. Storage management in single-threaded code is drudge work, but when concurrency is involved, it suddenly becomes a much thornier problem. People find themselves building their own slapdash garbage collection-like facilities to manage the lifetime of data objects that may be used in many different threads. These user-built facilities are tricky and often induce extra contention. Ultimately many concurrent algorithms, particularly lock-free ones, rely on the existence of a garbage collector (to prevent certain instances of ABA problems, for example*). Sure there are classes of techniques for lock-free reference counting (e.g. Detlefs et al.), but it seems like the modern mainstream languages with GC like Java and C# are gaining powerful high-level concurrent programming utilities a lot quicker than C and C++, and I suspect the higher difficulty of storage management is one influencing factor (obviously, the lack of a memory model and portability concerns are other major issues). Anyway, I mentioned to Bjarne that I thought the addition of garbage collection support was good because a lot of concurrent algorithms rely on it, and he said something to the effect of it being a crutch or somewhat lazy to rely on garbage collection just to avoid complicating concurrent algorithms with intricate reference counting. I don’t want to misrepresent his opinion, but that’s the general gist I got from his comment.

As far as I’m concerned, garbage collection is one of those largely inevitable evolutions of higher-level programming that is begrudged by some but eventually taken as a given. Some complained about overhead induced by operating systems when computers tended to run single applications; the same thing happened with the first compiled languages (versus hand-written assembly). Later it was VMs for managed languages or GC. Eventually, however, I think these will be taken for granted in mainstream programming. Java has already gone a long way to making garbage collection more of a necessary and mainstream feature. Now, certainly there are some gripes about the Java throwing out the baby with the bath water in not providing something as useful as explicitly scoped destruction to guarantee cleanup of certain related non-memory resources (like open files or sockets), and providing too weak semantics for finalize, but this isn’t a fundamental flaw of GC. Anyway, that’s tangential, but my view is that GC is nearly essential in new concurrent era, and I’m glad that C++0x is taking a first step in the right direction on that front.

Memory Model and Atomics
The rigorous memory model is, of course, also essential for great concurrent programming, and well known experts like Hans Boehm (and many others) have put a lot of work into that. I mentioned to Bjarne that I was also pleased that they are offering atomics and even support for relaxed consistency reads and writes useful in certain low-level concurrent algorithms. I told him that I thought that the inclusion of selective relaxed consistency, while definitely addressing only a limited audience of very skilled programmers, is really vindicated in light of Java’s experience — after initially resisting exposing such features, the developers of Java have come to the same conclusion that you need to expose these operations to allow the efficient implementation of certain kinds of concurrent data structures and algorithms. See Doug Lea’s work on the Java 7 Fences API and his announcement and rationale to the concurrency-interest list. He states:

The basic JSR133 JMM scheme provides three kinds of underlying ordering constaints, that are tied to variable declarations — read-volatile, write-volatile, and write-final (aka write-release, aka publish, aka lazy-set), or for stand-alone AtomicX objects.

This turns out not to mesh very well with the development of core concurrent algorithms (like the ones I tend to implement), where you often need these special flavors of reads and writes on an occasional basis, which is currently either impossible or insensible to achieve.
…
In the recent C++0x standard, this kind of usage was addressed by allowing per-usage modes on new read and write methods. (And further, supporting more than the three modes considered here, but I don’t think we need to introduce more for Java.)
…
So, after years of resisting the idea, my current conclusion is that we need to stop wishing for a miraculous solution to lack of call-by-ref, and instead allow developers to roll their own out of the raw ingredients — fences.

In his actual talk, Bjarne only briefly mentioned C++0x concurrency features and the memory model (because there was so much ground to cover), but after noting that “C++ 0x will have atomics,” he quipped that the new feature makes C++ a “superpower.” Besides the obvious pun, some people (myself included) would liken C++ to a nuclear weapon for a different reason — namely that it tends to provide many experts-only features with extremely intricate rules and many hidden pitfalls. This was illustrated perfectly to me by Bartosz Milewski’s attempt to implement a well-known synchronization algorithm (Peterson’s algorithm for shared-memory, two party mutual exclusion) using relaxed consistency atomics. Just by trying to implement Peterson’s algorithm, he ended up finding that the draft standard semantics were subtly broken, and it took Hans Boehm’s intervention and the modification of the standard just to get it to work. In his post titled The Inscrutable C++ Memory Model, he recounts just how complicated it is:

I had no idea what I was getting myself into when attempting to reason about C++ weak atomics. The theory behind them is so complex that it’s borderline unusable. It took three people (Anthony, Hans, and me) and a modification to the Standard to complete the proof of a relatively simple algorithm.

All the people involved are in the target population of expert users for relaxed consistency atomics, so the byzantine complexity illustrated here (i.e. “so complex that it’s borderline unusable”) doesn’t bode well for 99.9% of the C++ using population. Sometimes I’m afraid that some bold and foolhardy programmers won’t be able to resist the temptation of “optimizing” by using features like this — the programming equivalent of playing Russian Roulette with all chambers loaded. Of course, I still support their inclusion, since the Java experience shows that they are needed for a small subset of people like Doug Lea to write kickass foundational concurrent library functionality.

Anyway, to bring this post full circle, earlier I mentioned that shared-state concurrency can be tricky to apply to large-scale systems. One of the reasons is thematically similar to the “experts only” problem we just discussed — current abstractions for shared-state concurrency are very complex and tricky to get right. The STM people point out that locks don’t compose — in other words, to build on existing pieces and maintain invariants, you often have to “invite” other parties to participate in your own internal locking protocols, basically foiling true abstraction or encapsulation of concerns. Just how tricky are threads and locks? Well, Edward A. Lee’s “The Trouble With Threads” recounts the following anecdote:

A part of the Ptolemy Project experiment was to see whether effective software engineering practices could be developed for an academic research setting. We developed a process that included a code maturity rating system (with four levels, red, yellow, green, and blue), design reviews, code reviews, nightly builds, regression tests, and automated code coverage metrics [43]. The portion of the kernel that ensured a consistent view of the program structure was written in early 2000, design reviewed to yellow, and code reviewed to green. The reviewers included concurrency experts, not just inexperienced graduate students (Christopher Hylands (now Brooks), Bart Kienhuis, John Reekie, and myself were all reviewers). We wrote regression tests that achieved 100 percent code coverage. The nightly build and regression tests ran on a two processor SMP machine, which exhibited different thread behavior than the development machines, which all had a single processor. The Ptolemy II system itself began to be widely used, and every use of the system exercised this code. No problems were observed until the code deadlocked on April 26, 2004, four years later.

In closing, I just want to say I enjoyed Bjarne’s talk a lot, and I find him to be extremely pragmatic in technical matters. He’s very frank about C++’s purpose, the fact that it’s evolving and that it has certain warts (and also many things that are better than other languages). In other words, he’s not the Steve Jobs of C++, pretending it’s eternally perfect and consistent. There wasn’t much time for many questions, but the most amusing question to me was what can only be described as an “Oprah” question — “How do you feel about having your software running on so many systems?” Hah.

* See “ABA Prevention Using Single-Word Instructions” by Maged Michael and “DCAS is not a Silver Bullet for Nonblocking Algorithm Design” by Doherty et al. for some examples of ABA problems solved by GC and instances where GC alone is insufficient. However, regardless of whether GC solves the ABA problem entirely, GC tends to make concurrent algorithms significantly simpler in many other ways.