I lived there 3 yrs while a grad student and never had any problems. Worcester was my introduction to an American city. Clearly Clark and Columbia U, NYC share a location in an area with its share of social problems (the reason is the shift in relative wealth and poverty in that part of Worcester over the years) yet both are excellent schools nonetheless.
Much of the crime in Main South keeps out of the immediate University area, particularly the streets where the marvelous university owned houses are located, and does not touch students at all - it has its own social and economic dynamic that rarely bothers the students. Soem of Clark's best known profs live in Main South, and have done for years. But I never liked some of the streets half a mile to the north of campus, and for a time lived in the West of the city which is your average chocolate box New England suburbia - a different world. Dorms have excellent facilities and conviviality, and Clark's security and night-buses etc. are excellent. Speaking as a university professor - the undergrad eduation at Clark is some of the best in the land.