Mr Baker accused Theresa May of viewing her Liberal Democrat coalition colleagues as “a cuckoo in the nest rather than part of government”.
Speaking to the Independent newspaper, Mr Baker said: “They have looked upon it as a Conservative department in a Conservative government, whereas in my view it’s a Coalition department in a Coalition government.
“That mindset has framed things, which means I have had to work very much harder to get things done even where they are what the Home Secretary agrees with and where it has been helpful for the Government and the department.
“There comes a point when you don’t want to carry on walking through mud and you want to release yourself from that.”
The MP for Lewes was moved to the Home Office from Transport in October 2013.
Liberal Democrat leader Nick Clegg said he was “sorry” so see Mr Baker leave and described him as an “outstanding minister”.
In his formal resignation letter to the party leader, Mr Baker revealed that he had made clear in August that he was ready to quit – and pinned the blame squarely on his coalition colleagues.
“You will recognise that it has been particularly challenging being the only Lib Dem in the Home Office, which I see a newspaper the other day likened to being the only hippy at an Iron Maiden concert,” he said.
A Lib Dem spokesman said a replacement would be announced “in due course”.