In Gimp, I did the following:
- Created and filled a rectangle;
- Selected the Path tool;
- Clicked outside the bottom edge of the rectangle to start a path;
- Clicked outside the right edge of the rectangle to add another point;
- Clicked below and to the right of the rectangle to add another point;
- Ctrl-clicked on the start point to close the path, thereby forming a triangle across the corner of the rectangle;
- Edited the path segment that crossed the rectangle to make it nice and wavy;
- Used "Selection from path";
- Cut.
...and lo! I have a wavy-edged bit taken out of the rectangle
Was that what you wanted?