I am about half way through reading Remote – Office Not Required. The overall message is clear: you need an overlap of around four hours each working day for the whole team, and a comprehensive set of communication and collaboration tools. Beyond that, all inconveniences can be managed and the benefits substantially outweigh the difficulties.
This is exactly my own experience: you have to have sufficient time in the day to synchronise with everyone you work with, and this works fine using realtime collaboration tools such as Skype and Webex. To make the best use of this limited time, you also need a range of asynchronous tools, including document/code repositories, issue trackers and, (of course) email. When the team uses them effectively, they can be very productive, and what matters is the work produced.
The converse is also true: when everyone has to be on site every day, the measurement of performance becomes the number of hours present in the office. Not only is this counter-productive, but it is rather disconcerting to see dedicated professionals become clock-watchers.