John Rogers talks about Googles penalties on link exchanges. It is an interesting discussion in the comments. From what I have heard in the Search Engine Marketing forums, big link exchanges are a target with Google and getting them off your site is a good move.

However, keep your blogroll. Greg Swann makes a good argument of the non SEO benefits when putting together your blogroll.