In general, as long as the laser is well below 500mw, it is safe to look at the diffuse reflection. However, the goggles are there in case you accidentally hit a reflective surface. If you're using it without goggles, don't point it at glass, metal, or shiny plastic objects.
In addition, if...