Publish only changed files in IIS

When you use Visual Studio to republish files on IIS, it will only publish changed files by default.

All you need to do is to make sure that the option Remove additional files at destination is unchecked.

Reference:

https://social.msdn.microsoft.com/Forums/en-US/23f0486e-01c9-46eb-8488-79a6135a03c5/publish-only-changed-file?forum=aspconfiganddeploy

I was asking as it seem to take the same amount of time for any build and shows all the folders being published

Is that the normal process 

Yes this is the normal process.

From the published files you will be able to check the changed files using the Date modified.

Based on that Date modified copy the modified files and paste in to the server.

This way you can publish changes only.