These are decoration of photos. If you do not like this, just select No Decoration from the Decorate Photos With option. You can find this option above Theme panel.

If you are using Advanced mode, you can click Customize Properties button after you choose a theme. 

Then you can find an option called Decoration under Global tab. You just need to delete the file name of decoration if you don't need any decorations.