Ever hear of a "manual penalty" from Google? That could be the root of your problem.