Solaris cat is Super-Fast!!!

A colleague asked someone at work to do a test of disk performance by cat-ing a 32GB file to /dev/null to determine why we had slow backups.  It only took a fraction of a second - and he wondered why that could be the case.  

So I looked into it.  Firstly I used:

dd if=/path/to/largefile > /dev/null 

to see if it exhibited the same behaviour as cat.  It didn't.  Then I truss-ed both cat and dd to find out what the difference was.  I could see the data as an argument to the write system call in both processes,  but it turns out that cat uses mmap to map chunks of the file into the process address space rather than using the read system call.

So why does this make it really quick to read the file ?  Well, it doesn't.  I tried it this way:

cat /path/to/largefile | cat > /dev/null

Now it's much slower, the speed is far more in line with what you would expect for reading a large file from a disk array.

So what is going on ?  Well when you mmap a chunk of data into a process it's not really reading the data, it's just making it available to page in on demand.  When the only thing you do with that data pointer is pass it to the write system call, and write is pointing its output to /dev/null, the kernel is just throwing away the pointer.  Under normal circumstances if the data was written to stdout or another file it would be the write call that causes the data to be paged in - but since write is doing nothing, just returning, the data is never paged in.

But you say, if the data is not being read how could we have seen the data in the truss output ?  Well, truss is causing a small amount of each mmap-ed segment to be read from disk - a tiny amount compared to the size of the mmap-ed part - just enough for truss to read a few bytes from the start of each pointed to block, so truss is causing the kernel to page in this small amount of data because it is truss that wants to display it - if we weren't truss-ing the cat command the file wouldn't be read at all.  The fact that truss prints an exhaustive list of system calls used by cat to do this masks the slight slowdown that this small number of reads adds to the overall run-time of the cat command during truss-ing.  So take away truss and yes, the process is very nearly instant.