Logging

The logging facility in Sympathy is built on the standard logging module, see https://docs.python.org/3/howto/logging.html#logging-basic-tutorial for introductory material about it.

The level of logging used can be configured on the command line. There are three different ways to configure the logging, ranging from course to fine grained. Using the arguments –loglevel, –logger and –logging-config-file: these can not be mixed.

Logging output is normally written to the the standard error stream. On Windows, when running without a console, the output is redirected to a file called output.log - in the session folder.

The logging config file makes it possible to specify the configuration in a more detailed way and makes it possible to specify logging handlers to use.

Our top level logger, sympathy, and its child loggers (sympathy.core, sympathy.app, etc.) are used for all logging in our package. Third-party packages would use different loggers.

User logging

To use logging in user code, outside of the sympathy logger, simply use the logging module directly. Another option is to use sympathy.user logger, other loggers under sympathy for our internal use are not intended for user code. Use the sympathy.api.log.get_user_logger() function to get the user logger. One benefit with using this logger is that the log level can be configured using the basic configuration options.

Basic configuration

In the most basic case, where no configuration is made. The log level is simply set to ERROR for the root logger which is parent to sympathy and all other loggers.

The –loglevel option takes one argument which specifies the log level for the sympathy logger and makes it possible to set another level than ERROR, used for the root logger. For example: –loglevel warning.

The –logger option takes two arguments: the logger name and its log level. It can be repeated to specify the levels of multiple different loggers, for example, –logger app error and –logger core warning. The special logger name all can be used to configure the root logger and sympathy to configure our top level logger. Other logger names refer to loggers under sympathy, so app and core refers to sympathy.app and sympathy.core respectively.

Advanced configuration

For more advanced and detailed logging configuration, the –logging-config-file option is used to specify the dictionary used by logging.config.dictConfig (see https://docs.python.org/3/library/logging.config.html#configuration-functions) as a filename to a json file containing a dictionary.

Config file corresponding to no logging configuration:

{
    "version": 1,
    "formatters": {
	"basic": {
	    "()": "logging.Formatter",
	    "fmt": "%(levelname)s:%(name)s:%(message)s"
	}
    },
    "handlers": {
	"standard_output": {
	    "class": "logging.StreamHandler",
	    "formatter": "basic",
	    "stream": "ext://sys.stdout"
	}
    },
    "loggers": {
	"": {"level": "ERROR", "handlers": ["standard_output"]}
    }
}

Config file corresponding to –loglevel warning:

{
    "version": 1,
    "formatters": {
	"basic": {
	    "()": "logging.Formatter",
	    "fmt": "%(levelname)s:%(name)s:%(message)s"
	}
    },
    "handlers": {
	"standard_output": {
	    "class": "logging.StreamHandler",
	    "formatter": "basic",
	    "stream": "ext://sys.stdout"
	}
    },
    "loggers": {
	"": {"level": "ERROR", "handlers": ["standard_output"]},
	"sympathy": {"level": "DEBUG", "handlers": ["standard_output"], "propagate": 0}
    }
}

Config file corresponding to –logger app error and –logger core warning:

{
    "version": 1,
    "formatters": {
	"basic": {
	    "()": "logging.Formatter",
	    "fmt": "%(levelname)s:%(name)s:%(message)s"
	}
    },
    "handlers": {
	"standard_output": {
	    "class": "logging.StreamHandler",
	    "formatter": "basic",
	    "stream": "ext://sys.stdout"
	}
    },
    "loggers": {
	"": {"level": "ERROR", "handlers": ["standard_output"]},
	"sympathy.app": {"level": "ERROR", "handlers": ["standard_output"], "propagate": 0},
	"sympathy.core": {"level": "WARNING", "handlers": ["standard_output"], "propagate": 0}
    }
}

It is possible to do much more with the config file. Formatters, handlers and loggers can be configured allowing logging output to different locations and in different formats. Included sympathy.api.log.handlers.LockFileHandler can be used to safely log output to a single file.

Config file which outputs sympathy logger in debug mode to lock file:

{
    "version": 1,
    "formatters": {
	"basic": {
	    "()": "logging.Formatter",
	    "fmt": "%(levelname)s:%(name)s:%(message)s"
	}
    },
    "handlers": {
	"standard_output": {
	    "class": "sympathy.api.log.handlers.LockFileHandler",
	    "formatter": "basic",
	    "lock_filename": "ABSOLUTE-PATH-TO-LOCK-FILE.lock",
	    "filename": "ABSOLUTE-PATH-TO-LOG-FILE.txt",
	    "timeout": 0.5
	}
    },
    "loggers": {
	"sympathy": {"level": "DEBUG", "handlers": ["standard_output"]}
    }
}

Before trying, make sure to change lock_filename and filename to where you want the output.

It is also possible to use other formatting. Included sympathy.api.log.formatters.JsonFormatter will output each log entry in JSON format:

{
    "version": 1,
    "formatters": {
	"json": {
	    "()": "sympathy.api.log.formatters.JsonFormatter",
	    "fields": {
		"levelname": "level",
		"name": "logger",
		"message": "message",
		"exc_text": "exception",
		"stack_info": "stack",
		"asctime": "time"
	    }
	}
    },
    "handlers": {
	"standard_output": {
	    "class": "logging.StreamHandler",
	    "formatter": "json",
	    "stream": "ext://sys.stdout"
	}
    },
    "loggers": {
	"sympathy": {"level": "DEBUG", "handlers": ["standard_output"]}
    }
}

Also included are two handlers for sending log output as JSON over HTTP: sympathy.api.log.handlers.JsonBodyHTTPHandler will post the entire log record converted to JSON and sympathy.api.log.handlers.FormatJsonBodyHTTPHandler allows the conversion to be configured with the JsonFormatter.

Config file which sends json formatted records over HTTP:

{
    "version": 1,
    "formatters": {
	"json": {
	    "()": "sympathy.api.log.formatters.JsonFormatter",
	    "fields": {
		"levelname": "level",
		"name": "logger",
		"message": "message",
		"exc_info": "exception",
		"stack_info": "stack",
		"asctime": "time"
	    }
	}
    },
    "handlers": {
	"http_output": {
	    "class": "sympathy.api.log.handlers.FormatJsonBodyHTTPHandler",
	    "formatter": "json",
	    "host": "localhost:9022",
	    "url": "/",
	    "method": "post"
	}
    },
    "loggers": {
	"sympathy": {"level": "INFO", "handlers": ["http_output"]}
    }
}

Other handlers from logging can also be used, for example, logging.handlers.SocketHandler which pickles the log record and sends it over a TCP socket.

Sending logging output to a server (HTTP or socket) can be used as an alternative to the LockFileHandler for safely logging output from different processes. Here is an example using the SocketHandler from the logging documentation: https://docs.python.org/3/howto/logging-cookbook.html#sending-and-receiving-logging-events-across-a-network.