It seems to me that the danger is that unmoderated content, and content that you have no control over, can appear to be associated with your site. At least in a normal wiki there is an opportunity for moderation or repair of vandalism by the community but not, as I understand it, with sidewiki.

If this came from anyone but Google I think it would be easy to ignore it and watch it fade away. But, unfortunately, as it comes from Google there is a frighteningly good chance it may be with us for a while.

It also means that site owners would feel the need to install the wretched Google toolbar just to monitor a sidewiki.