WikiEditor

Nice try, but I wasn't able to get it to work (mostly because of my cluelessness). I eventually found and adapted another user's custom js as follows:

// Listes de caractères
var api_liste_fr = ["ɑ","ɑ̃","ə","ɛ","œ","ø","ɛ̃","ɔ","ɔ̃","œ̃","ʃ","ɡ","ɥ","ʒ","ɲ","ŋ","ʁ","‿","ː","ˈ","ˌ"] ;
var caracteres_liste_fr = ["’","æ","Æ","€","ñ","œ","Œ","–","—","…"] ;

// Ajout de la section Caractères du Wiktionnaire > API français
hookEvent( 'load', function() {
	$j( '#wpTextbox1' ).wikiEditor( 'addToToolbar', {
		'sections': {
			'caracteres': {
				label:'Mes outils',
				type:'booklet',
				deferLoad:true,
				pages: {
					'API_fr': {
						'label':'API français',
						'layout':'characters',
						'characters': api_liste_fr
					},
					'car_fr': {
						'label':'Caractères spéciaux',
						'layout':'characters',
						'characters': caracteres_liste_fr
					}
				}
			}
		}
	} );
} );

That’ll get you a separate booklet, while the code I suggested adds pages to the existing Special Characters booklet. It probably only works when you put it in a site-wide script like MediaWiki:Common.js/edit.js, rather than in one of your user scripts.

Unfortunately, I can’t test either version at the French Wiktionary because the use of mw.toolbar.insertButton() in MediaWiki:Common.js/edit.js (under “Boutons depuis MW 1.18”) has completely broken the WikiEditor toolbar.