system-wide: fix escaping of magic chars

* fix escaping of magic lua chars (#2800)
* fix redundant second gsub line in shellstartsqescape function
* fix return value of shellstartsqescape function

Signed-off-by: Dirk Brenken <dev@brenken.org>