John Lennon recalled glazed eyes and giggling on Dick Lester's zany Bond spoof – now recognised as the film that invented pop video, as well as the moment The Beatles' songwriting left Merseybeat far behind. Mark Beaumont on how a classic was created out of chaos