Jason you missed one very significant cause of buffering. Improper encoding was the cause in my case. None of my mp4 files (1080P, 720P) would stream without buffering every 15 seconds and using the cpu at 100%. The videos had been created with either GoPro Studio or Adobe Premiere Elements. To fix the issue I used HandBrake to re-encode the files with these changes,

Video
H.264 Profile : High
H.264 Level : 5.0

Audio
Codec : HE-AAC

Now they stream perfectly with no buffering. The cpu use hovers around 4%.