SDL_VideoModeOK returns 0 if the requested mode is not supported under any bit depth, or returns the bits-per-pixel of the closest available mode with the given width, height and requested surface flags (see SDL_SetVideoMode).
The bits-per-pixel value returned is only a suggested mode. You can usually request and bpp you want when setting the video mode and SDL will emulate that color depth with a shadow video surface.
The arguments to SDL_VideoModeOK are the same ones you would pass to SDL_SetVideoMode