There are several ways to create custom widgets for RAP. This article is not a guide on how to create them, but only links to the relevant resources and adds covers some current issues.
SWT-Style custom widgets
Those are custom widgets like they are created in SWT. There are two types: Compound widgets and Self-drawing widgets.
This guide by the creators of SWT pre-dates RAP, but is still valid for current SWT/RAP versions. The chapters about native-code can be ignored.
The current RAP developers guide covers some RAP-specific aspects of such custom-widgets.