Introduction
The go
programming language’s
profiling system
supports an
interesting feature: http access. If your application already
has an HTTP interface, and it uses the net/http default multiplexer
(i.e., it calls http.Handle() and http.ListenAndServe()), enabling http
profiling is as simple as:
import _ "net/http/pprof"
and the profiling APIs will be visible under /debug/pprof/ alongside
your application’s URL paths, where you can access them with:
go tool pprof http://host:port/debug/pprof/profile
While this is a slick way of profiling a running
program, the fact that this capability is installed by
an import makes turning it off and on cumbersome: the
import either has to be added and removed, or placed
in a separate file with
build tags
to turn compilation
on and off. If left in shipped code, it has the side effect
of publishing the profiling hooks (with no access control)
on the application http listener. This is far from ideal,
but there are a few good workarounds.
Access Control
A simple and effective option is to put the pprof http server on a separate port on localhost, separate from the application http server. If the application does not use the http default multiplexer, starting the profiling http server is as simple as:
go func() {
log.Fatal(http.ListenAndServe("localhost:8081"))
}()
Otherwise, you can run the following additional setup prior to any http.Handle() calls:
pprofMux := http.DefaultServeMux
http.DefaultServeMux = http.NewServeMux()
go func() {
log.Fatal(&http.Server{
Addr: "localhost:8081",
Handler: pprofMux,
}.ListenAndServe())
}()
Either of the above snippets can be conditionally run, so the profiling server may be turned on or off by command line flags or application configuration. However, since they are now only locally accessible, there is no downside to leaving them on.
Remote Control
Now that you have the profiling hooks safely exposed via a localhost-only interface, you can invoke the following:
go tool pprof http://localhost:8081/debug/pprof/profile
and profile the program with the local go tools. However, since go compiles to a static binary which can be installed without any go-related dependencies, chances are you don’t have go tools where your program is running. But you probably have SSH, so:
ssh -L 8081:localhost:8081 user@remote-host
or if you are already logged in:
~C -L 8081:localhost:8081
will make the local go tool pprof on your local machine profile
the remote code over the ssh tunnel.
Conclusion
If you are running go services on servers with only trusted local users, exposing the profiling http port through port forwarding limits access to the trusted local users, making it practical to leave profiling on as just another administrative feature of the system. This allows the developer to observe and/or troubleshoot the program under a real-world, and real, load.
Chris Mikkelson is a Senior Distributed Systems Engineer at Farsight Security, Inc.
