I'm not quite sure what you mean here, but all the background images that I have used in my website are not
square (mostly being 3:2 ratio). If you mean non-rectilinear, then I suspect not, since as far as I know all gif, jpeg, tiff, etc. files are rectangles.
You can fake it, of course, using The Gimp or Photoshop and making a border with an irregular inner contour around your picture. If you make it the same color as the overall background color that you set the theme, or make it transparent, people won't be able to see it.
Flieg (just a simple user -- do not ask me for CSS or PHP advice.

)