With Yeovil 2-0 up courtesy of Olufela Olomola and Otis Khan, the hosts netted twice in the second period to level matters.
Bailey admitted to being at fault for Chesterfield’s first and atoned for his error by heading home only the fourth goal of his career in injury time.
Town’s skipper told iFollow Glovers: “There’s nothing better than scoring a goal let alone a last minute winner away.
“I need to watch it back but the first goal looked like it was my fault as I got caught ball-watching a bit so thankfully I made up for it with the last minute goal.
“I don’t know what I was doing up there! I’m usually more defensive but the gaffer has given me a bit more of a license to get forward, get shots off and nick a few goals.
“I asked the ref how long was left, he said ‘three minutes’ so I thought I’d give it one last run.
“I looked up and saw Sam (Surridge) and no one was there so I ran into the space and thankfully he picked me out.
“It was my first ever header – luckily it came off my forehead and not the side of my head!”
Bailey’s effort secured Yeovil’s first away win of the season and first three points since March.
A lot has been spoken about that record but the ex-Coventry City man revealed the squad “always believed in ourselves”.
He added: “The players always stuck together and knew it was a matter of time.
“When it went to 2-2 I thought ‘here we go again’ but we showed a bit of grit and determination and pulled it back.”