From Quake Wiki
Vanilla Quake is a commonly used term that refers to Quake without any modifications running. It generally used in the map making community as a way to indicate that a level is intended to be run with the standard Quake gameplay and no additional monsters, weapons or other enhancements. It's also a term used often when talking about map based hacks, which are things that can be done in standard Quake that were never intended by id Software, often involving the use of the info_notnull entity.
Technically speaking, vanilla Quake refers to Quake running the game from the ID1 folder (-game ID1).