To manufacture a magnetic memory element by a self-aligned forming method without using the lithography in a nonvolatile magnetic memory manufacturing method.

The method of manufacturing a nonvolatile magnetic memory with magnetic memory elements 3 comprises a step of forming a surface-exposed conductor 51 on a first insulation film 41 formed on a substrate, removing the top of the conductor 51 to form recesses 52 in the insulation film 41 on the conductor 51, forming a magnetic memory element film 61 for forming the magnetic memory elements 3 and a cap film 37 on the insulation film 41 including the insides of the recesses 52 so as to have depressions 38 on the cap film 37 upside on the conductor 51, forming a second insulation film 42 in the depressions 38, and removing the cap layer 37 and the memory element film 61 with the second insulation film 42 used as a mask to form the magnetic memory elements 3 capped with the cap film 37 from the memory element film 61 left in the recesses 52.