Falling out of favour with The Blades, he initially joined City on loan in February 1983. He quickly took Denis Smith's place in the side, a month later, the loan deal was made permanent, the latest in a long line of imposing free transfer signings from Sheffield United.

He was one of the best centre halves to play for City, being quick and commanding, both strong on the tackle and in the air, a true leader, both on and off the pitch. In the air, he never missed a header and his distribution was second to none. A popular captain and big influence on the team, he was voted City's Clubman Of The Year in both 1984 and 1985 (some feat with Keith Walwyn and John Byrne around). In those seasons, City set a new Football League record of 101 points in a season and then beat Arsenal and took Liverpool to an FA Cup replay. Undoubtedly, as club captain and with his "twin", Alan Hay to the fore, they ensured a tremendous team spirit was built. Tony Canham described Macphail as "defender, leader and he could play".

In total, he played 173 games for City, scoring 29 goals, most coming from set pieces or the penalty spot.

In 1986, after his City contract expired and he was unable to agree a new deal, he joined Bristol City for a tribunal determined £14,000 fee. A year later, after talk of a return to City fell through, he signed for Denis Smith's Sunderland in 1987, helping them to 2 promotions back into the top flight where he played one game before moving onto Hartlepool, initially as a player and later manager. He was to leave the game and in 2019 was believed to be a car salesman in Durham.

Off the pitch, a misdemeanour earned him a place in City's Naughty Boys list.

Writing in the Roker Report (23^rd March 1988), Martin Wanless recalled.

When those of a certain vintage look back at solid, reliable central defenders, John MacPhail is one who springs to mind. The 32 year-old Scotsman, who Smith had managed at York City before signing him for Sunderland from Bristol City, was a tricky player to pigeonhole.

On the one hand, he was a tough, rugged, no-nonsense defender. On the other, he had a reasonable turn of pace, was comfortable on the ball, and scored a decent amount of goals, too. Since arriving for the unusual fee of £23,000, MacPhail became a mainstay in defence, a reliable partner for the talented but sometimes wayward Gary Bennett.

Smith was reminiscing on his first impressions of MacPhail, as the duo prepared for Saturday's return to Bootham Crescent. The John MacPhail that goes back to York City on Saturday is unrecognisable from the man who walked into Bootham Crescent five years ago. 'He was a nightmare when I first saw him'. Smith then recalled a scouting mission to The Hawthorns to watch MacPhail, then of Sheffield United, line up for their stiffs against West Brom. A scouting mission that his wife Kate accompanied him to. Must have been Denis' choice of a romantic night out. 'When I first went to watch him in a reserve game for Sheffield United at West Brom, I brought my wife Kate along. When I told her who I was watching even she told me she didn't rate him. He had ability but no idea of how to play centre half. He didn't have any direction or confidence. I think that John would agree that I rescued his career. He is a changed man, both on and off the field, He has been a very good influence in the dressing room and I don't think I could have said that four or five years ago. He is maybe getting a little bit long in the tooth to play in the First Division, but I have no doubt that he is now capable of holding his own in the top flight'.

MacPhail served Smith well during his time at Roker Park. In the three full seasons he was at the club he enjoyed promotion twice, captained the side for a spell during 1988-89 when Smith took the captaincy away from Gary Bennett and, at the age of 35, fulfilled Denis Smith's prophecy, he did play in the top flight. As Sunderland started the season in the First Division away at Norwich, MacPhail was handed a start at the expense of new summer signing Kevin Ball. If it had been anyone but Denis Smith picking the team you may have suspected it was a sentimental selection, but in reality, it was anything but. Bally had struggled in pre-season, and MacPhail's form had deserved, what would turn out to be, his only appearance in the top division. It resulted in a 3-2 defeat, and for the home visit of Spurs a few days later, MacPhail was dropped in favour of Ball. Sunderland played out a 0-0 defeat, keeping Gary Lineker and Paul Gascoigne, fresh from Italia 90, at bay, and that was it for MacPhail's Sunderland career.

After 153 appearances and 22 goals, 'Monty' was off to Hartlepool, where he wrapped up his playing career, and enjoyed his only stint in management.

In December 2021, John MacPhail suffered a fall which resulted in life changing brain injuries. In October 2022, a just giving page had been set up to raise funds to allow significant adaptions to his family home to allow him to return home.