PURPOSE: To provide an attaching structure of a tool which enables easy attachment and removal of a tool such as a collet to and from a tool holder.

CONSTITUTION: A collet attaching hole 8 is formed in a collet attaching part 6 of a collet holder and a stopping pin 15 which is attached to pass through in a radius direction is fixed to the collet attaching part 6. A cross section of the stopping pin 15 is circular and a part thereof is arranged to project inside the collet attaching hole 8 of the collet attaching part 6. When the collet 9 is inserted, the stopping pin elastically deforms externally in a radius direction due to a tapered part of an end part thereof, and the deformed part is contained in a relief recessed part 16 formed in a rear of the stopping pin 15. When the collet 9 is moved, the deformed part of the stopping pin 15 returns to its original state and enters a cut-out part 17 formed in an outer peripheral surface of an attaching part 10 of the collet 9 to stop the collet 9.