1. Set the build to enable parallel running and use multiple MSBuild processes say 4 processes. /m:4 /p:BuildInParallel=true
Setting /m /p:BuildInParallel=true is better to let the number of MSBuild instances auto decide depending on build agent capability. More info http://www.hanselman.com/blog/FasterBuildsWithMSBuildUsingParallelBuildsAndMulticoreCPUs.aspx
With this set, the build server (agent) is still taking considerable time, and build only reduced by by maximum 5 minutes. In the build agent noticed Antimalware protection is running, taking lot of CPU, while the build is going on.