It is important to realise the difference between Closing and Deleting a Web Part. By default the Reminder Web Parts will not let you Close them for reasons explained below. Therefore you must delete them form a page instead.

Deleting a Web Part

To Delete a web part you must first Select Modify Shared Page > Design This Page
You can then select Delete from the Web Parts Menu Note - Reminder Web Parts will not have a Close menu item for reasons described below.

Closing a Web Part

By default SharePoint Reminder Web Parts will not allow you to close them, although you can change this. Closed web parts are often mistaken for Deleted Web Parts. If you Close rather than Delete a web part you will receive a warning in your servers event log.

You can then delete the web part by Adding it back onto the page and then deleting it or by using the Web Part Maintenance page.

Deleting a closed web part using the Web Part Maintenance page

You can open the Web Part Maintenance view of any web part page by appending ?contents=1 onto the end of the URL? E.g.

http://www.yourserver.com/yoursite/yourpage.aspx?contents=1

The following page will open

You can the web part that has been closed (Open on page = No) and delete it by selecting the checkbox and pressing Delete

Adding a closed web part back onto a page in order to Delete it

You can select the Close command from a Web Parts Menu.
The web part is not actually deleted from the page, it is removed from the Visible area of the page and placed in the Web Part Page Gallery. To Delete this web part completely you would drag it back onto the page and perform the steps for Deleting A Web Part above.