If you have a custom action under SharePoint 2010 “welcome menu” or under “site actions menu” and when you click on the custom action you want to open the application page in standard SharePoint modal dialog. Here is the solution,

You need to change the UrlAction code in the element file as given below

<?xml version="1.0" encoding="utf-8"?> 

<Elements xmlns="http://schemas.microsoft.com/sharepoint/"> 
  <CustomAction 
        Id ="CustomMenu_MoreInformation"
        GroupId="PersonalActions"
        Location="Microsoft.SharePoint.StandardMenu"
        Sequence="1000"
        Title="More Information"
        Description="View more information"  
        ImageUrl="_layouts/ebloginImages/logo.png">
    <UrlAction Url="javascript:OpenPopUpPageWithTitle('{SiteUrl}/SitePages/MoreInformatin.aspx, RefreshOnDialogClose, 600, 400,'My Custom Action')"/>

  </CustomAction>
</Elements>

Now when you click on the menu option, your application page will open in modal dialog as shown in below pictures.