Answer: Programmatically, set the VendorShell resource XmNmwmDecorations to 0,
such as:

  #include <Xm/MwmUtil.h> /* see MWM_DECOR_* and MWM_FUNC_* */
  #include <Xm/DialogS.h>
  popupShell =
      XtVaCreatePopupShell( "PopupShell",
                            xmDialogShellWidgetClass, toplevel,
                            XmNmwmDecorations, 0,
                            NULL );

With the 0, you have no decorations at all, but if you want just a little
frame, use MWM_DECOR_BORDER instead.

Thanks to Guillaume.Gallais@asm.thomson.fr for the code fragment and pointing
out that there is no MWM_DECOR_NONE.

Reinhard M. Weiss (weissrm@execpc.com) also pointed out that MWM_DECOR_NONE
was fictitious. He also added:

"I have found that the resource XtNoverrideRedirect does cause the olwm to
remove all decorations (my guess is that it would work in mwm roughly the
same).  This works programmatically as well as in resource files (i.e.
*.className*overrideRedirect: true). There are some undesirable effects to
this, however, particularly with focus and managing dialogs and popups."