The verb to buttonhole not only means to make buttonholes in a garment, but also to attract the attention of someone and detain them in conversation against their will.
This verb appears to be an altered form of to button-hold, meaning to take hold of someone by a button and detain them in conversation against their will. To button-hold is implied in the earlier agent noun button-holder, first attested in The Miseries of Human Life; or The Groans of Timothy Testy, and Samuel Sensitive (1806), by the British writer and clergyman James Beresford (1764-1840); Dialogue the Seventh, titled Miseries of Social Life, contains the following:
At your own house—being kept half an hour without your hat in a drizzling rain, in attending a button-holder to your gate.
As such, the verb to button-hold is first recorded in The Examiner (London) of 20th April 1817:
ENGRAVINGS RECENTLY PUBLISHED
Raimbach’s ‘Rent-Day’, from Wilkie. A large print from a first-rate painting. [...] In considering this work of Mr. Raimbach, let us try to suppress the thousand thoughts and emotions respecting the politically oppressed and their oppressors, which so immediately arise from the first glance at the ‘Rent-Day’, where a throng of heart-aching tenants are coming to pay, and to excuse themselves for not paying, their rent [...].
The reader who has seen Mr. Wilkie’s picture [...] will smilingly recollect [...] the face-distorted, cork-screw pulling, butler,—the button-holding conversation of the two defaulters.
The verb to button-hole is first attested in The Dublin Evening Post of 16th May 1846:
New Orleans, April 21, 1846.—The city was thrown into a feverish state of excitement on learning that [...] Mexico had declared war against the United States, and that our naval force had, in consequence, blockaded her ports. The rumour obtained very general credit, and nothing else was talked of for hours and hours, and the arrival of the ‘Colonel Harney’ was anxiously looked for. About eleven o’clock she came crawling along, and the newspaper people soon boarded and button-holed every man or boy who would be likely to know anything at all about passing events.
Interestingly, The Limerick Reporter of 19th May of that year used the form button-held when reporting on the same event:
The newspaper people [...] button-held every man or boy who would be likely to know anything at all about passing events.
Similarly, an article titled Buttons, published in the magazine All The Year Round of 28th June 1862, contains the following:
And there is the man who is button-holed, or held, poor wretch! and must listen to half an hour’s harangue about nothing interesting, while his friends are waiting dinner, or his wife is sitting in her diamonds and opera cloak, sullenly expecting his escort. The man who button-holes another is a ruffian, not fit for civilised society, and ought to go out to the long-winded savages who have not yet learnt that brevity is the soul of wit.
This shows that the verb to button-hold and its altered form to button-hole were both in usage at the time. Perhaps the reason for this alteration is that in the spoken language button-hold was mistaken for a past tense or a past participle, which led to the verb to button-hole being invented in order to match the original error.
Robert Hendrickson gave a comical explanation in The Facts on File: Encyclopedia of Word and Phrase Origins (4th edition, 2008):
In those days men’s coats had buttons all the way up to the neck, including one on the lapel that could be buttoned in cold weather. When fashion decreed that upper buttons be eliminated, button-holders didn’t suddenly reform. Instead, they began grabbing people by the buttonholes designers (for no good reason) left on the lapels and the phrase became ‘to buttonhole’.
It is typical of folk etymologies that stories have to be made up in order to support false theories.