In the past my father owned a sign manufacturing business. We used silicone to glue letters and signs to concrete, brick stucco etc... when doing installations. Silicone is very tough once dried, but can be removed easily enough, when you want to take the sign down. Other adhesives such as liquid nails, can be very difficult to clean once dried. One additional suggestion, I know you don't want to drill any holes, however if you drill two holes one at each end in the morter, and silicone in threaded metal bolts (studs) the holes can be easily patched when the sign is removed.
Hope this helps.