I have a system that creates files on disk, then uses mmap and madvise and mflush to asynchronously do I/O to the disk. This system may potentially create many, many files, each of which will have three mmap sections, that will be rotated through the file.
After trying to run this system for a while, I started getting segmentation violations that I couldn't quite understand. Initially, I thought it was a threading problem, because I'm using boost::asio and boost::thread quite heavily. I used strace() to figure out what the system was doing, and found that right before the crashes, one or more calls to mmap() would fail.
Long story short: There is a limit to the number of mmap() segments that can be active in a Linux process at any one time. This limit is configurable in /proc/sys/vm/max_map_count. I already knew there was a file descriptor limit, and I raised that pretty high, but apparently Linux doesn't think you'll be using lots of mmap() just because you're using lots of files. Adding the following to /etc/rc.local will fix the problem:
echo 1000000 > /proc/sys/vm/max_map_count