Or it can be your boss.
New research from the Harvard Business School and Stanford University found that a bad boss can make you physically ill, almost as if you smoke a pack of cigarettes a day, according to Travis Bradberry, co-founder of job recruiting website TalentSmart.
The research, which analyzed more than 200 studies on stress in the workplace, found that worrying about getting fired or laid off can make you 50 percent more likely to face health issues. An overly demanding job also made people 35 percent more likely to get sick, Bradberry wrote. Most of these negative health effects are comparable to ones that cigarette smokers face.
And it's these bad bosses who either demand too much of their employees or put their employees in constant fear that they will lose their job.
The report found that 75 percent of workers say their boss is their biggest stress at work, but 59 percent won't leave that manager behind, BBC reported, which can also be a problem.
"The problem is that the longer you stay with an abusive boss, the harder it becomes to convince yourself to leave and the bigger the toll it takes on your mind, body, and family," Bradberry wrote. "That's why it's important to know how to recognize the signs of a bad boss early on, before you're sucked in so deeply that it's hard to leave."
But this isn't exactly a new finding. Researchers have been saying for years that a bad boss can affect one's mental and physical health.
Jonathan D. Quick, a medical instructor at Harvard Medical School, told The Washington Post in 2014 that bad bosses can make your work life hard and cause you to have physical and mental illnesses, like heart attacks, high blood pressure, sleep problems and anxiety.
Bad bosses can also lead employees to pick up bad habits like smoking, excessive alcohol use and overeating, Quick told The Post.
"The evidence is clear that the leadership qualities of 'bad' bosses over time exert a heavy toll on employees' health," Quick told The Post. "The evidence is also clear that despite the rationalizations some leaders may use to defend their stress-inducing, unsupportive style, such behavior by leaders does not contribute to improved individual performance or organizational productivity."
Part of the problem is that bad bosses don't know they're being bad bosses, Amy Rees Anderson wrote for Forbes. Signs of a bad boss include a lack of vision for the company, they only communicate with a select group of workers, they aren't happy with their job and they don't try to inspire employees.
Bad bosses are also known to accept mediocre work. Rather than pushing employees to provide something better, they reward employees based on "brownnosing" rather than skill and they attack specific people rather than employee performance, she wrote.
"You see the difficulty is that most of us judge ourselves by our intentions, while others judge us based on our behavior," Anderson wrote. "So the fact is that most bosses may feel they have perfectly good intentions and therefore they justify or disregard their own bad behavior and as a result employees are left to suffer the effects of dealing with a bad boss."
Bad bosses may want to increase communication with employees and develop healthy relationships with them to fix some of these issues, Ray Williams wrote for Psychology Today.
"With tough economic times already producing considerable stress for employees in terms of increased workloads and expectations for greater performance, the physical and mental health of employees is of paramount importance," Williams wrote. "A key strategy for addressing this issue is ensuring that executives and managers have positive and healthy relationships with employees. The cost is too high to endure any longer, 'bad bosses.'"
But what can the employee do, especially when they can't leave their job?
Anderson suggests focusing on their bosses' good qualities to make the job seem less stressful. Employees may also want to try behaving like a good leader so that the "bad boss" can learn from it. It may also be important for a worker to use their time at work as a way to learn lessons about what not to do as a boss, Anderson wrote.
"Try to learn the 'what not to do's' from your bad boss until you can work for a good one," Anderson wrote. "Chances are that one day you will be a boss, too, and any lessons you learn now will help you to behave better when the mantle of leadership is placed on your shoulders."