Running mode and edit mode
When a dialog opens, it is in running mode, the mode for normal use. In that mode, you can use the dialog but you cannot bring changes to it such as adding a new button.
The Edit ▸ Edit mode ⌘Y menu item toggles a dialog into "edit mode". In edit mode you can bring any change to the dialog. Furthermore, in edit mode the Action menu of the menu bar gives access to the tools required in the development of custom dialogs.
You know that a dialog is in edit mode when it displays a resize handle in the bottom right corner.
Each step in the process of creating a custom dialog is described below.
Once you are familiar with making a regular custom dialog, you may want to use some of the advanced features available.