A REST API is an API that meets the requirements for RESTful architecture, which include (among other things):
Independent client and server applications
Statelessness (requests must include all information necessary for processing)
Cacheability
Layered architecture
REST APIs are meant to be more scalable and more lightweight than some other API designs, and RESTful architecture is thus a popular choice for modern API architectures.
A serverless REST API is just a REST API that has been deployed using a serverless architecture. For example, you could build a serverless REST API using AWS Lambda for serverless functions and CockroachDB Serverless as your serverless database.