From Quake Wiki
QBSP is the first stage of map compiling. It is the process that reads the map file and generates the bsp file, and in doing so calculates all of the geometry and collision data for the level. It also extracts any needed textures from the associated wad file (or files) and repackages them into the bsp file. After this step, the level can be loaded into the game engine. However, it will not have any lighting or visibility data until you run Light and Vis.
What QBSP Does
Using the brushes from the map file's worldspawn entity, it calculates the bsp tree that stores all visible surfaces in the game. It also generates the collision hulls, which are also bsp trees but do not have any visible surfaces, and are only used for collision detection. For each additional entity that has brushes, it generates a submodel which also has its own collision hulls. Any textures needed by the level are copied into the bsp file from the associated wad files. At runtime, the engine will load textures from the bsp, not the wad files.
QBSP also generates a prt file containing the portal information necessary for Vis to run. The prt file is not needed or used by the game engine. If a map has a leak, QBSP will generate a pointfile instead of a prt file. This file can be used by the game engine to display a visual trail of particles showing you where the leak is. If a map has multiple leaks, QBSP will only find the first one.
QBSP is a command-line tool, generally named qbsp.exe. You can invoke it from a command prompt, via a shortcut or batch file, or from the "compile" or "build" menus of some level editors.
Error Messages and Warnings