The Joys of DirectIO
It’s been a long time since I’ve had to do any serious C programming, but sometimes you need to get close to the metal.
We’ve been having an issue with log files poisoning our file cache by sheer volume of useless pages being written and saved in cache therefore pushing good pages out of cache. Yes we have fairly verbose log files and yes we can change the log level, but we have so many different log files being written it’s kind of pointless.
So after a little research I came across DirectIO which is writing directly to the disk bypassing the file cache. Now before I go any further for the Linus fanboys/purists I tried doing all this with posix_fadvise and unfortunately it doesn’t seem to be honoured on Fedora Core 5, such the pity.
So off to scraping the metal with O_DIRECT, which after looking at few web pages seemed like this was going to be a cake walk, unfortunately that wasn’t my experience. There are also a dearth of complete examples on how to do direct IO. So. You can have the source if it will help you, there are a number of issues I ran across like have to align buffers in memory for DirectIO. You can get a tar of the source here.
I apologize for how ugly this code snipit looks but TinyMCE insists on reformatting any variation of raw HTML I try to use, a fix is in the works for this one.
/*****
* write a page to memory, these pages require aligning the in memory buffer for
* writing directly to the disk
*/
void bufwrite( char *buff, int fd )
{
char *buf_unaligned = malloc((PSIZE * sizeof(char)) + PSIZE - 1);
char *buf_aligned = (char *) (((unsigned long)buf_unaligned + PSIZE - 1) & (~(PSIZE-1)));
memcpy( buf_aligned, buf, PSIZE );
int result = write( fd, buf_aligned, PSIZE );
if( result < PSIZE ) {
fprintf( stderr, "\nbufwrite:Error bufwriting result = %d offset = %d\n", result, offset );
perror( " bufwrite:" );
}
free( buf_unaligned );
}
I think the better solution is to have a dedicated partition marked as DirectIO, but it would appear in Fedora that you will
need to install a newer file system like ReiserFS or ZFS (as far as I can tell)
Other possible useful directIO links: