Some logging documentation (#6498)
parent
8fcbfa7df9
commit
059b52759e
@ -0,0 +1,396 @@ |
||||
--- |
||||
date: "2019-04-02T17:06:00+01:00" |
||||
title: "Advanced: Logging Configuration" |
||||
slug: "logging-configuration" |
||||
weight: 55 |
||||
toc: true |
||||
draft: false |
||||
menu: |
||||
sidebar: |
||||
parent: "advanced" |
||||
name: "Logging Configuration" |
||||
weight: 55 |
||||
identifier: "logging-configuration" |
||||
--- |
||||
|
||||
# Logging Configuration |
||||
|
||||
The logging framework has been revamped in Gitea 1.9.0. |
||||
|
||||
## Log Groups |
||||
|
||||
The fundamental thing to be aware of in Gitea is that there are several |
||||
log groups: |
||||
|
||||
* The "Default" logger |
||||
* The Macaron logger |
||||
* The Router logger |
||||
* The Access logger |
||||
* The XORM logger |
||||
* A logger called the `GitLogger` which is used during hooks. |
||||
|
||||
There is also the go log logger. |
||||
|
||||
### The go log logger |
||||
|
||||
Go provides its own extremely basic logger in the `log` package, |
||||
however, this is not sufficient for our purposes as it does not provide |
||||
a way of logging at multiple levels, nor does it provide a good way of |
||||
controlling where these logs are logged except through setting of a |
||||
writer. |
||||
|
||||
We have therefore redirected this logger to our Default logger, and we |
||||
will log anything that is logged using the go logger at the INFO level. |
||||
|
||||
### The "Default" logger |
||||
|
||||
Calls to `log.Info`, `log.Debug`, `log.Error` etc. from the `code.gitea.io/gitea/modules/log` package will log to this logger. |
||||
|
||||
You can configure the outputs of this logger by setting the `MODE` |
||||
value in the `[log]` section of the configuration. |
||||
|
||||
Each output sublogger is configured in a separate `[log.sublogger]` |
||||
section, but there are certain default values. These will not be inherited from the `[log]` section: |
||||
|
||||
* `FLAGS` is `stdflags` (Equal to |
||||
`date,time,medfile,shortfuncname,levelinitial`) |
||||
* `FILE_NAME` will default to `%(ROOT_PATH)/gitea.log` |
||||
* `EXPRESSION` will default to `""` |
||||
* `PREFIX` will default to `""` |
||||
|
||||
The provider type of the sublogger can be set using the `MODE` value in |
||||
its subsection, but will default to the name. This allows you to have |
||||
multiple subloggers that will log to files. |
||||
|
||||
### The "Macaron" logger |
||||
|
||||
By default Macaron will log to its own go `log` instance. This writes |
||||
to `os.Stdout`. You can redirect this log to a Gitea configurable logger |
||||
through setting the `ENABLE_MACARON_REDIRECT` setting in the `[log]` |
||||
section which you can configure the outputs of by setting the `MACARON` |
||||
value in the `[log]` section of the configuration. `MACARON` defaults |
||||
to `file` if unset. |
||||
|
||||
Each output sublogger for this logger is configured in |
||||
`[log.sublogger.macaron]` sections. There are certain default values |
||||
which will not be inherited from the `[log]` or relevant |
||||
`[log.sublogger]` sections: |
||||
|
||||
* `FLAGS` is `stdflags` (Equal to |
||||
`date,time,medfile,shortfuncname,levelinitial`) |
||||
* `FILE_NAME` will default to `%(ROOT_PATH)/macaron.log` |
||||
* `EXPRESSION` will default to `""` |
||||
* `PREFIX` will default to `""` |
||||
|
||||
NB: You can redirect the macaron logger to send its events to the gitea |
||||
log using the value: `MACARON = ,` |
||||
|
||||
### The "Router" logger |
||||
|
||||
There are two types of Router log. By default Macaron send its own |
||||
router log which will be directed to Macaron's go `log`, however if you |
||||
`ENABLE_MACARON_REDIRECT` you will enable Gitea's router log. You can |
||||
disable both types of Router log by setting `DISABLE_ROUTER_LOG`. |
||||
|
||||
If you enable the redirect, you can configure the outputs of this |
||||
router log by setting the `ROUTER` value in the `[log]` section of the |
||||
configuration. `ROUTER` will default to `console` if unset. The Gitea |
||||
Router logs the same data as the Macaron log but has slightly different |
||||
coloring. It logs at the `Info` level by default, but this can be |
||||
changed if desired by setting the `ROUTER_LOG_LEVEL` value. |
||||
|
||||
Each output sublogger for this logger is configured in |
||||
`[log.sublogger.router]` sections. There are certain default values |
||||
which will not be inherited from the `[log]` or relevant |
||||
`[log.sublogger]` sections: |
||||
|
||||
* `FILE_NAME` will default to `%(ROOT_PATH)/router.log` |
||||
* `FLAGS` defaults to `date,time` |
||||
* `EXPRESSION` will default to `""` |
||||
* `PREFIX` will default to `""` |
||||
|
||||
NB: You can redirect the router logger to send its events to the Gitea |
||||
log using the value: `ROUTER = ,` |
||||
|
||||
### The "Access" logger |
||||
|
||||
The Access logger is a new logger for version 1.9. It provides a NCSA |
||||
Common Log compliant log format. It's highly configurable but caution |
||||
should be taken when changing its template. The main benefit of this |
||||
logger is that Gitea can now log accesses in a standard log format so |
||||
standard tools may be used. |
||||
|
||||
You can enable this logger using `ENABLE_ACCESS_LOG`. Its outputs are |
||||
configured by setting the `ACCESS` value in the `[log]` section of the |
||||
configuration. `ACCESS` defaults to `file` if unset. |
||||
|
||||
Each output sublogger for this logger is configured in |
||||
`[log.sublogger.access]` sections. There are certain default values |
||||
which will not be inherited from the `[log]` or relevant |
||||
`[log.sublogger]` sections: |
||||
|
||||
* `FILE_NAME` will default to `%(ROOT_PATH)/access.log` |
||||
* `FLAGS` defaults to `` or None |
||||
* `EXPRESSION` will default to `""` |
||||
* `PREFIX` will default to `""` |
||||
|
||||
If desired the format of the Access logger can be changed by changing |
||||
the value of the `ACCESS_LOG_TEMPLATE`. |
||||
|
||||
NB: You can redirect the access logger to send its events to the Gitea |
||||
log using the value: `ACCESS = ,` |
||||
|
||||
#### The ACCESS_LOG_TEMPLATE |
||||
|
||||
This value represent a go template. It's default value is: |
||||
|
||||
`{{.Ctx.RemoteAddr}} - {{.Identity}} {{.Start.Format "[02/Jan/2006:15:04:05 -0700]" }} "{{.Ctx.Req.Method}} {{.Ctx.Req.RequestURI}} {{.Ctx.Req.Proto}}" {{.ResponseWriter.Status}} {{.ResponseWriter.Size}} "{{.Ctx.Req.Referer}}\" \"{{.Ctx.Req.UserAgent}}"` |
||||
|
||||
The template is passed following options: |
||||
|
||||
* `Ctx` is the `macaron.Context` |
||||
* `Identity` is the `SignedUserName` or `"-"` if the user is not logged |
||||
in |
||||
* `Start` is the start time of the request |
||||
* `ResponseWriter` is the `macaron.ResponseWriter` |
||||
|
||||
Caution must be taken when changing this template as it runs outside of |
||||
the standard panic recovery trap. The template should also be as simple |
||||
as it runs for every request. |
||||
|
||||
### The "XORM" logger |
||||
|
||||
The XORM logger is a long-standing logger that exists to collect XORM |
||||
log events. It is enabled by default but can be switched off by setting |
||||
`ENABLE_XORM_LOG` to `false` in the `[log]` section. Its outputs are |
||||
configured by setting the `XORM` value in the `[log]` section of the |
||||
configuration. `XORM` defaults to `,` if unset, meaning it is redirected |
||||
to the main Gitea log. |
||||
|
||||
XORM will log SQL events by default. This can be changed by setting |
||||
the `LOG_SQL` value to `false` in the `[database]` section. |
||||
|
||||
Each output sublogger for this logger is configured in |
||||
`[log.sublogger.xorm]` sections. There are certain default values |
||||
which will not be inherited from the `[log]` or relevant |
||||
`[log.sublogger]` sections: |
||||
|
||||
* `FILE_NAME` will default to `%(ROOT_PATH)/xorm.log` |
||||
* `FLAGS` defaults to `date,time` |
||||
* `EXPRESSION` will default to `""` |
||||
* `PREFIX` will default to `""` |
||||
|
||||
### The Hook and Serv "GitLoggers" |
||||
|
||||
These are less well defined loggers. Essentially these should only be |
||||
used within Gitea's subsystems and cannot be configured at present. |
||||
|
||||
They will write log files in: |
||||
|
||||
* `%(ROOT_PATH)/hooks/pre-receive.log` |
||||
* `%(ROOT_PATH)/hooks/update.log` |
||||
* `%(ROOT_PATH)/hooks/post-receive.log` |
||||
* `%(ROOT_PATH)/serv.log` |
||||
* `%(ROOT_PATH)/http.log` |
||||
|
||||
In the future these logs may be rationalised. |
||||
|
||||
## Log outputs |
||||
|
||||
Gitea provides 4 possible log outputs: |
||||
|
||||
* `console` - Log to `os.Stdout` or `os.Stderr` |
||||
* `file` - Log to a file |
||||
* `conn` - Log to a keep-alive TCP connection |
||||
* `smtp` - Log via email |
||||
|
||||
Certain configuration is common to all modes of log output: |
||||
|
||||
* `LEVEL` is the lowest level that this output will log. This value |
||||
is inherited from `[log]` and in the case of the non-default loggers |
||||
from `[log.sublogger]`. |
||||
* `STACKTRACE_LEVEL` is the lowest level that this output will print |
||||
a stacktrace. This value is inherited. |
||||
* `MODE` is the mode of the log output. It will default to the sublogger |
||||
name. Thus `[log.console.macaron]` will default to `MODE = console`. |
||||
* `COLORIZE` will default to `true` for `file` and `console` as |
||||
described, otherwise it will default to `false`. |
||||
|
||||
### Non-inherited default values |
||||
|
||||
There are several values which are not inherited as described above but |
||||
rather default to those specific to type of logger, these are: |
||||
`EXPRESSION`, `FLAGS`, `PREFIX` and `FILE_NAME`. |
||||
|
||||
#### `EXPRESSION` |
||||
|
||||
`EXPRESSION` represents a regular expression that log events must match to be logged by the sublogger. Either the log message, (with colors removed), must match or the `longfilename:linenumber:functionname` must match. NB: the whole message or string doesn't need to completely match. |
||||
|
||||
Please note this expression will be run in the sublogger's goroutine |
||||
not the logging event subroutine. Therefore it can be complicated. |
||||
|
||||
#### `FLAGS` |
||||
|
||||
`FLAGS` represents the preceding logging context information that is |
||||
printed before each message. It is a comma-separated string set. The order of values does not matter. |
||||
|
||||
Possible values are: |
||||
|
||||
* `none` or `,` - No flags. |
||||
* `date` - the date in the local time zone: `2009/01/23`. |
||||
* `time` - the time in the local time zone: `01:23:23`. |
||||
* `microseconds` - microsecond resolution: `01:23:23.123123`. Assumes |
||||
time. |
||||
* `longfile` - full file name and line number: `/a/b/c/d.go:23`. |
||||
* `shortfile` - final file name element and line number: `d.go:23`. |
||||
* `funcname` - function name of the caller: `runtime.Caller()`. |
||||
* `shortfuncname` - last part of the function name. Overrides |
||||
`funcname`. |
||||
* `utc` - if date or time is set, use UTC rather than the local time |
||||
zone. |
||||
* `levelinitial` - Initial character of the provided level in brackets eg. `[I]` for info. |
||||
* `level` - Provided level in brackets `[INFO]` |
||||
* `medfile` - Last 20 characters of the filename - equivalent to |
||||
`shortfile,longfile`. |
||||
* `stdflags` - Equivalent to `date,time,medfile,shortfuncname,levelinitial` |
||||
|
||||
### Console mode |
||||
|
||||
For loggers in console mode, `COLORIZE` will default to `true` if not |
||||
on windows, or the windows terminal can be set into ANSI mode or is a |
||||
cygwin or Msys pipe. |
||||
|
||||
If `STDERR` is set to `true` the logger will use `os.Stderr` instead of |
||||
`os.Stdout`. |
||||
|
||||
### File mode |
||||
|
||||
The `FILE_NAME` defaults as described above. If set it will be relative |
||||
to the provided `ROOT_PATH` in the master `[log]` section. |
||||
|
||||
Other values: |
||||
|
||||
* `LOG_ROTATE`: **true**: Rotate the log files. |
||||
* `MAX_SIZE_SHIFT`: **28**: Maximum size shift of a single file, 28 represents 256Mb. |
||||
* `DAILY_ROTATE`: **true**: Rotate logs daily. |
||||
* `MAX_DAYS`: **7**: Delete the log file after n days |
||||
* NB: `COLORIZE`: will default to `true` if not on windows. |
||||
* `COMPRESS`: **true**: Compress old log files by default with gzip |
||||
* `COMPRESSION_LEVEL`: **-1**: Compression level |
||||
|
||||
### Conn mode |
||||
|
||||
* `RECONNECT_ON_MSG`: **false**: Reconnect host for every single message. |
||||
* `RECONNECT`: **false**: Try to reconnect when connection is lost. |
||||
* `PROTOCOL`: **tcp**: Set the protocol, either "tcp", "unix" or "udp". |
||||
* `ADDR`: **:7020**: Sets the address to connect to. |
||||
|
||||
### SMTP mode |
||||
|
||||
It is not recommended to use this logger to send general logging |
||||
messages. However, you could perhaps set this logger to work on `FATAL`. |
||||
|
||||
* `USER`: User email address to send from. |
||||
* `PASSWD`: Password for the smtp server. |
||||
* `HOST`: **127.0.0.1:25**: The SMTP host to connect to. |
||||
* `RECEIVERS`: Email addresses to send to. |
||||
* `SUBJECT`: **Diagnostic message from Gitea** |
||||
|
||||
## Default Configuration |
||||
|
||||
The default empty configuration is equivalent to: |
||||
|
||||
```ini |
||||
[log] |
||||
ROOT_PATH = %(GITEA_WORK_DIR)/log |
||||
MODE = console |
||||
LEVEL = Info |
||||
STACKTRACE_LEVEL = None |
||||
REDIRECT_MACARON_LOG = false |
||||
ENABLE_ACCESS_LOG = false |
||||
ENABLE_XORM_LOG = true |
||||
XORM = , |
||||
|
||||
[log.console] |
||||
MODE = console |
||||
LEVEL = %(LEVEL) |
||||
STACKTRACE_LEVEL = %(STACKTRACE_LEVEL) |
||||
FLAGS = stdflags |
||||
PREFIX = |
||||
COLORIZE = true # Or false if your windows terminal cannot color |
||||
``` |
||||
|
||||
This is equivalent to sending all logs to the console, with default go log being sent to the console log too. |
||||
|
||||
## Log colorization |
||||
|
||||
Logs to the console will be colorized by default when not running on |
||||
Windows. Terminal sniffing will occur on Windows and if it is |
||||
determined that we are running on a terminal capable of color we will |
||||
colorize. |
||||
|
||||
Further, on *nix it is becoming common to have file logs that are |
||||
colored by default. Therefore file logs will be colorised by default |
||||
when not running on Windows. |
||||
|
||||
You can switch on or off colorization by using the `COLORIZE` value. |
||||
|
||||
From a development point of view. If you write |
||||
`log.Info("A %s string", "formatted")` the `formatted` part of the log |
||||
message will be Bolded on colorized logs. |
||||
|
||||
You can change this by either rendering the formatted string yourself. |
||||
Or you can wrap the value in a `log.ColoredValue` struct. |
||||
|
||||
The `log.ColoredValue` struct contains a pointer to value, a pointer to |
||||
string of bytes which should represent a color and second set of reset |
||||
bytes. Pointers were chosen to prevent copying of large numbers of |
||||
values. There are several helper methods: |
||||
|
||||
* `log.NewColoredValue` takes a value and 0 or more color attributes |
||||
that represent the color. If 0 are provided it will default to a cached |
||||
bold. Note, it is recommended that color bytes constructed from |
||||
attributes should be cached if this is a commonly used log message. |
||||
* `log.NewColoredValuePointer` takes a pointer to a value, and |
||||
0 or more color attributes that represent the color. |
||||
* `log.NewColoredValueBytes` takes a value and a pointer to an array |
||||
of bytes representing the color. |
||||
|
||||
These functions will not double wrap a `log.ColoredValue`. They will |
||||
also set the ResetBytes to the cached resetBytes. |
||||
|
||||
Be careful not to change the contents of resetBytes or boldBytes as this |
||||
will break rendering of logging elsewhere. You have been warned. |
||||
|
||||
## Log Spoofing protection |
||||
|
||||
In order to protect the logs from being spoofed with cleverly |
||||
constructed messages. Newlines are now prefixed with a tab and control |
||||
characters except those used in an ANSI CSI are escaped with a |
||||
preceding `\` and their octal value. |
||||
|
||||
## Creating a new named logger group |
||||
|
||||
Should a developer wish to create a new named logger, `NEWONE`. It is |
||||
recommended to add an `ENABLE_NEWONE_LOG` value to the `[log]` |
||||
section, and to add a new `NEWONE` value for the modes. |
||||
|
||||
A function like `func newNewOneLogService()` is recommended to manage |
||||
construction of the named logger. e.g. |
||||
|
||||
```go |
||||
func newNewoneLogService() { |
||||
EnableNewoneLog = Cfg.Section("log").Key("ENABLE_NEWONE_LOG").MustBool(false) |
||||
Cfg.Section("log").Key("NEWONE").MustString("file") // or console? or "," if you want to send this to default logger by default |
||||
if EnableNewoneLog { |
||||
options := newDefaultLogOptions() |
||||
options.filename = filepath.Join(LogRootPath, "newone.log") |
||||
options.flags = "stdflags" |
||||
options.bufferLength = Cfg.Section("log").Key("BUFFER_LEN").MustInt64(10000) |
||||
generateNamedLogger("newone", options) |
||||
} |
||||
} |
||||
``` |
||||
|
||||
You should then add `newOneLogService` to `NewServices()` in |
||||
`modules/setting/setting.go` |
Loading…
Reference in new issue