HTTP client patterns in Go form the backbone of modern network applications. I’ll share my experience implementing these patterns in production environments, focusing on practical examples that ensure reliable communication.
The foundation begins with proper client configuration. In Go, the http.Client offers extensive customization options. Here’s how I typically set up a production-ready client:
client := &http.Client{
Timeout: time.Second * 10,
Transport: &http.Transport{
MaxIdleConns: 100,
MaxConnsPerHost: 20,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
DisableCompression: false,
DialContext: (&net.Dialer{
Timeout: 5 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
},
}
Request customization is crucial for handling authentication, headers, and context. I’ve found this pattern particularly effective:
func createRequest(ctx context.Context, method, url string, body io.Reader) (*http.Request, error) {
req, err := http.NewRequestWithContext(ctx, method, url, body)
if err != nil {
return nil, fmt.Errorf("create request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("User-Agent", "MyApp/1.0")
return req, nil
}
Response handling requires careful attention to prevent memory leaks and ensure proper resource cleanup:
func handleResponse(resp *http.Response) ([]byte, error) {
if resp == nil {
return nil, fmt.Errorf("nil response")
}
defer resp.Body.Close()
body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if err != nil {
return nil, fmt.Errorf("read body: %w", err)
}
if resp.StatusCode >= 400 {
return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode, string(body))
}
return body, nil
}
I’ve implemented robust retry mechanisms that handle transient failures gracefully:
func retryableClient(maxRetries int, backoffFactor float64) *RetryClient {
return &RetryClient{
client: http.DefaultClient,
maxRetries: maxRetries,
backoffFactor: backoffFactor,
}
}
type RetryClient struct {
client *http.Client
maxRetries int
backoffFactor float64
}
func (rc *RetryClient) Do(req *http.Request) (*http.Response, error) {
var resp *http.Response
var err error
for attempt := 0; attempt <= rc.maxRetries; attempt++ {
if attempt > 0 {
delay := time.Duration(float64(time.Second) * math.Pow(rc.backoffFactor, float64(attempt-1)))
time.Sleep(delay)
}
reqCopy := req.Clone(req.Context())
resp, err = rc.client.Do(reqCopy)
if err == nil && resp.StatusCode < 500 {
return resp, nil
}
}
return resp, fmt.Errorf("max retries exceeded: %w", err)
}
Connection pooling optimizes resource usage and improves performance. Here’s my preferred configuration:
func createPooledClient() *http.Client {
transport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
MaxIdleConns: 100,
MaxIdleConnsPerHost: 10,
MaxConnsPerHost: 20,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
return &http.Client{
Transport: transport,
Timeout: 30 * time.Second,
}
}
Rate limiting is essential for respecting API limits and maintaining good citizenship:
type RateLimitedClient struct {
client *http.Client
limiter *rate.Limiter
}
func NewRateLimitedClient(rps float64) *RateLimitedClient {
return &RateLimitedClient{
client: http.DefaultClient,
limiter: rate.NewLimiter(rate.Limit(rps), 1),
}
}
func (rlc *RateLimitedClient) Do(req *http.Request) (*http.Response, error) {
err := rlc.limiter.Wait(req.Context())
if err != nil {
return nil, fmt.Errorf("rate limit: %w", err)
}
return rlc.client.Do(req)
}
Error handling is crucial for maintaining system stability. I implement comprehensive error types:
type HTTPError struct {
StatusCode int
Message string
URL string
}
func (e *HTTPError) Error() string {
return fmt.Sprintf("HTTP %d: %s (URL: %s)", e.StatusCode, e.Message, e.URL)
}
func checkResponse(resp *http.Response) error {
if resp.StatusCode >= 400 {
return &HTTPError{
StatusCode: resp.StatusCode,
Message: http.StatusText(resp.StatusCode),
URL: resp.Request.URL.String(),
}
}
return nil
}
Context management ensures proper timeout handling and cancellation:
func fetchWithContext(url string) ([]byte, error) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return nil, fmt.Errorf("create request: %w", err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, fmt.Errorf("do request: %w", err)
}
defer resp.Body.Close()
return io.ReadAll(resp.Body)
}
Circuit breakers prevent cascading failures:
type CircuitBreaker struct {
client *http.Client
failures int
threshold int
timeout time.Duration
lastError time.Time
mu sync.Mutex
}
func (cb *CircuitBreaker) Do(req *http.Request) (*http.Response, error) {
cb.mu.Lock()
if cb.failures >= cb.threshold && time.Since(cb.lastError) < cb.timeout {
cb.mu.Unlock()
return nil, fmt.Errorf("circuit breaker open")
}
cb.mu.Unlock()
resp, err := cb.client.Do(req)
if err != nil {
cb.mu.Lock()
cb.failures++
cb.lastError = time.Now()
cb.mu.Unlock()
return nil, err
}
cb.mu.Lock()
cb.failures = 0
cb.mu.Unlock()
return resp, nil
}
These patterns form a comprehensive toolkit for building reliable network applications in Go. The key is combining them effectively based on specific requirements while maintaining simplicity and reliability.
Remember to implement proper logging, metrics collection, and monitoring to maintain visibility into your application’s network behavior. This ensures quick problem identification and resolution in production environments.
Through careful implementation of these patterns, you can build robust, efficient, and maintainable network applications that handle real-world challenges effectively.
