Two interfaces carry almost the entire standard library, and each of them has exactly one method. A reader hands you bytes. A writer takes them. That is the whole contract, and once I stopped thinking of it as a small detail of Go and started thinking of it as the actual shape of I/O, a lot of code that used to feel heavy became light.

package main

import (
	"io"
	"os"
)

func main() {
	// Every source and sink in the standard library speaks this dialect.
	var r io.Reader = os.Stdin
	var w io.Writer = os.Stdout

	if _, err := io.Copy(w, r); err != nil {
		panic(err)
	}
}

Files, sockets, gzip streams, hash functions, HTTP request bodies, and cloud storage objects all line up behind those two shapes. Because they do, any one of them can be swapped for any other and the code around the swap does not notice. That is the real payoff. You write a function that takes an io.Reader, and later you can feed it a file, a network connection, or a string in a test without touching the function.

The interesting work is not in writing your own reader or writer. It is in joining them. The io package and a few friends around it are full of small connectors, and the nine that follow are the ones my fingers reach for without thinking. I will show each one, explain what it does in plain language, and point at the trap hiding inside it, because most of these have one.

Buffering is the first habit to build, and understanding why takes a moment. When you call Read on an os.File, you are making a system call. A system call is your program asking the kernel to do something, and the round trip is expensive relative to almost anything else you do in a loop. If you read a one-megabyte file one byte at a time, you make roughly a million system calls, and your program crawls. The fix is to read a big chunk once and then hand out bytes from that chunk in memory.

func readFirstLine(path string) (string, error) {
	f, err := os.Open(path)
	if err != nil {
		return "", err
	}
	defer f.Close()

	r := bufio.NewReaderSize(f, 64*1024)
	line, err := r.ReadString('\n')
	if err != nil && line == "" {
		return "", err
	}
	return strings.TrimRight(line, "\r\n"), nil
}

Notice the error handling on the read. ReadString returns whatever it managed to read plus an error. If the last line of a file has no trailing newline, you get the line and io.EOF together. Dropping the line in that case loses real data. So I check whether anything came back before I treat the error as fatal. That one detail has bitten me more than once in log processors.

For line-oriented work, bufio.Scanner is the friendlier tool. It splits on a delimiter, it is fast, and the API is tiny. The catch is that it carries a default token ceiling of sixty-four kilobytes. Feed it a single log line longer than that and it stops with bufio.ErrTooLong. This is not a bug, it is a guard against a malicious stream eating all your memory in one line. Call Scanner.Buffer with a bigger maximum before the first Scan if your input has unpredictable line lengths.

func scanLongLines(r io.Reader) error {
	s := bufio.NewScanner(r)
	s.Buffer(make([]byte, 0, 1024*1024), 8*1024*1024)

	for s.Scan() {
		process(s.Bytes())
	}
	return s.Err()
}

The s.Err() at the end matters more than it looks. A scanner that stops because of an I/O failure looks exactly like a scanner that finished normally, unless you ask. I once spent an afternoon chasing a missing batch of records that turned out to be a truncated read nobody had checked.

Buffering helps on the way out too. bufio.Writer collects small writes and flushes them in one system call when the buffer fills or when you call Flush. The classic mistake is forgetting to flush before the process exits, or ignoring the error that Flush returns. A Flush failure means bytes never left your program, and if you are writing to a file or a socket, that error is the only signal you will get.

When several sources should behave like one, io.MultiReader does the joining. It reads the first reader until it is exhausted, then moves to the next, in order. This is a clean way to layer defaults underneath user configuration, so that a missing config file simply means the defaults stand alone.

func layeredConfig(path string) io.Reader {
	defaults := strings.NewReader("port=8080\nworkers=4\n")

	f, err := os.Open(path)
	if err != nil {
		return defaults
	}
	return io.MultiReader(defaults, f)
}

Order sets precedence only if the parser you use lets later keys win. Many do, but not all, and I have seen a YAML library quietly keep the first value it saw. Check yours before you rely on the arrangement. Errors surface in the same order they are read, so a failure in the defaults reader means the file is never touched at all, which is usually what you want but is worth knowing.

io.MultiWriter runs the mirror trick on the output side. A single Write call gets duplicated to every writer in the list, which makes it trivial to send a log stream to standard output and a file at the same time.

func openLog(path string) (io.Writer, func(), error) {
	f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
	if err != nil {
		return nil, nil, err
	}
	return io.MultiWriter(os.Stdout, f), func() { f.Close() }, nil
}

One detail catches people. MultiWriter stops at the first writer that returns an error and skips the rest for that call. So a partial failure leaves some destinations updated and others untouched, and the caller sees a single error with no clue about which writer accepted the bytes. If your writers carry independent weight, like a billing ledger next to an audit log, handle each one separately. Bundling them is convenient right up until it costs you money.

Reading and watching at the same time is what io.TeeReader gives you. Every byte pulled through the tee is also written to a second destination, which is exactly how you compute a checksum over data you are streaming somewhere else without buffering the whole thing in memory.

func uploadWithDigest(r io.Reader, dst io.Writer) (string, error) {
	h := sha256.New()
	tee := io.TeeReader(r, h)

	if _, err := io.Copy(dst, tee); err != nil {
		return "", err
	}
	return hex.EncodeToString(h.Sum(nil)), nil
}

The tee only writes as fast as the reader is drained, and it does no buffering of its own. Hash functions never fail, so this example is safe. Tee into a file or a network writer instead and that writer’s errors come back out of Read, which means the copy loop above already routes them somewhere sensible. I use this shape for object storage uploads where I want the checksum handed to the server in the same request.

io.Pipe is the connector for goroutines. It hands back a matched pair where writes to one end block until the other end is read. That gives you a synchronous, in-memory channel that satisfies the Reader and Writer interfaces, so you can run a compressor, an encryptor, or any streaming transform in a separate goroutine and present the result as a plain reader.

func compressedStream(payload []byte) io.Reader {
	pr, pw := io.Pipe()

	go func() {
		zw := gzip.NewWriter(pw)
		_, err := zw.Write(payload)
		if cerr := zw.Close(); err == nil {
			err = cerr
		}
		pw.CloseWithError(err)
	}()

	return pr
}

Close pipes with CloseWithError, not a bare Close. A plain close tells the reader the stream ended cleanly, which is indistinguishable from success, and the consumer will happily treat a half-written stream as complete. Passing the error through preserves the reason the producer stopped. The other thing to remember is that the writer blocks until someone reads, so if the consumer walks away early, your goroutine sits there forever. When the caller might abandon the stream, give the goroutine a way out through a context or a done channel.

Bounding input is a defensive habit worth adopting wherever data arrives from outside your process. io.LimitReader caps how many bytes a consumer can pull from an underlying source. The HTTP standard library gives you the same idea in a friendlier wrapper called http.MaxBytesReader, which also slams the connection shut when the limit is hit.

func readCapped(r io.Reader, max int64) ([]byte, error) {
	data, err := io.ReadAll(io.LimitReader(r, max))
	if err != nil {
		return nil, err
	}

	if int64(len(data)) == max {
		var probe [1]byte
		if n, _ := r.Read(probe[:]); n > 0 {
			return nil, fmt.Errorf("input exceeds %d bytes", max)
		}
	}
	return data, nil
}

Here is the subtlety. A limit reader truncates silently. It reports io.EOF once the allowance runs out, exactly as it would for a stream that genuinely finished. If you need to reject oversized input rather than quietly accept its first slice, probe the underlying reader for one extra byte after you hit the limit, the way I do above. It is one read and it turns a silent trim into an honest error.

Custom readers interpose your own logic in the middle of a chain. A counter, a decompressor, a decryptor, a line splitter, all follow the same shape. You receive a buffer from the caller, fill it from the source, and adjust the count if you transformed anything along the way.

type countingReader struct {
	r io.Reader
	n int64
}

func (c *countingReader) Read(p []byte) (int, error) {
	n, err := c.r.Read(p)
	c.n += int64(n)
	return n, err
}

func (c *countingReader) BytesRead() int64 { return c.n }

Respect the contract when you write one of these, because the contract is smaller than it looks and easier to break than it seems. You may return a non-zero count together with a non-nil error, and callers must handle both values. You should never return zero and a nil error, because that tells the caller to keep asking forever. Return the source’s error untouched so io.EOF keeps its meaning all the way up the chain. Discarding the count when an error appears loses bytes; swallowing the error hides the end of the stream. I have written both mistakes, and both took longer to find than they took to make.

Copying is where the fast paths live, and they are easy to break by accident. io.Copy does not just loop over a buffer. Before it does anything, it checks whether the source implements WriterTo or the destination implements ReaderFrom. When it finds one, it hands the whole transfer to that method. For a TCP connection feeding a file, that can mean the kernel moves the bytes itself with sendfile or splice, never bringing them into your program’s memory at all. Only when both checks fail does the plain buffer loop run.

var copyBufPool = sync.Pool{
	New: func() any {
		b := make([]byte, 32*1024)
		return &b
	},
}

func copyWithBuffer(dst io.Writer, src io.Reader) (int64, error) {
	bufp := copyBufPool.Get().(*[]byte)
	defer copyBufPool.Put(bufp)
	return io.CopyBuffer(dst, src, *bufp)
}

Reach for io.CopyBuffer when you want control over memory, like capping the working set on a small host, or when you want to reuse buffers under heavy throughput. It still checks the same fast paths before it uses your buffer, so you are not trading speed for a fixed allocation. If the buffer you handed in is never touched, the fast path won, and that is the outcome to hope for. Wrap your reader in something that hides WriterTo and you lose the shortcut without knowing it, which is a quiet way to make a service slower after a refactor that looked harmless.

Random access inside a stream is what io.SectionReader offers. It presents a window over a larger ReaderAt, defined by an offset and a length, and it refuses to read past its own boundary. Seek positions are measured from the start of the window, not from the start of the underlying data.

func readRecord(r io.ReaderAt, offset, length int64) ([]byte, error) {
	section := io.NewSectionReader(r, offset, length)
	return io.ReadAll(section)
}

Since os.File implements ReaderAt, this maps onto pread semantics and never moves the shared file offset. That makes it safe for many goroutines to read different regions of the same file at once, each with its own window and no locking between them. It is the natural way to build an index over a large flat file, or to hand a parser exactly the byte range it needs and nothing more. I have used it to serve fixed-size records out of a file far bigger than memory, and the code around it stayed tiny.

These connectors stack in any order you like, and the order changes the meaning. A limit wrapped outside a decompressor caps the compressed bytes coming in. Wrap it inside instead and it caps the decompressed bytes going out, which is usually what you actually meant. A tee placed before a buffer sees many small reads. Place it after and it sees the larger ones. Neither is wrong, but they are different programs.

So sketch the chain on paper before you write the code. Draw arrows and label each one with what it does to the bytes passing through. Once the picture is right, the Go writes itself, because every one of these connectors takes an io.Reader and returns an io.Reader, or takes an io.Writer and returns an io.Writer. That is the trick the whole package is built on, and it is why two one-method interfaces can hold up an entire language’s worth of I/O.

The last thing I would tell anyone learning this is to keep error handling visible at every join. Streaming code fails in the middle, not at the start, and the errors are the only evidence you get. Check the count and the error together. Close the pipes with the real reason. Flush the writers before you walk away. None of it is clever, and all of it is the difference between a pipeline that works and one that works until the day it quietly does not.