You can delete a picture or video by going to My Stuff, then selecting the My Photos or My Videos folder.
There you can either select a group of items to delete by touching the small red trash can in the lower right-hand corner of the thumbnail image, then touching the red trash can button at the bottom of the screen.
You will then be prompted to confirm that you'd like to permanently delete the item(s).
You can also delete an item that's viewed in full screen by selecting the red trash can button.