The neighbourhood I grew up in was diverse and full of working-class pride, but all that its critics could see was crime and isolation