Take a look at Task Manager on the machine where your TFS Repository lives.  We have this problem whenever its memory utilization is greater than 95%.

http://stackoverflow.com/questions/20632215/tfs-build-controller-service-http-code-500-system-servicemodel-serviceactivatio

TFS Build Controller service HTTP code 500: System.ServiceModel.ServiceActivationException