Golang Internals Part 1: Autogenerated functions (and how to get rid of them)
Maybe if you are like us at MinIO, you have every now and then come across ‘autogenerated’ functions in your Golang call stacks and wondered what they are all about?
We had a case the other day where the call stack showed something like the following:
cmd.retryStorage.ListDir(0x12847c0, 0xc420402e70, 0x1, ...)
minio/cmd/retry-storage.go:183 +0x72
cmd.(*retryStorage).ListDir(0xc4201624b0, 0xdf3b5f, 0xe, 0x25, ...)
:932 +0xaf
cmd.cleanupDir.func1(0xf18f1ec540, 0x25, 0x24, 0xde7eb5)
minio/cmd/object-api-com.go:215 +0xe1
cmd.cleanupDir(0x1284860, 0xc4201624b0, 0xdf3b5f, ...)
minio/cmd/object-api-com.go:231 +0x1d1
As you can see the second function looks similar to the first function (value receiver, defined in our MinIO code at minio/cmd/retry-storage.go:183), however it takes a pointer receiver instead. Also suspiciously there is just a line number shown for the second function but the file name for the source code is absent.
The code that created this call stack originates from the following function where a (recursive) function is declared that in turn calls storage.ListDirwhereby storage is passed into cleanupDir.
// Cleanup a directory recursively.
func cleanupDir(storage StorageAPI, volume, dirPath string) error {
var delFunc func(string) error
// Function to delete entries recursively.
delFunc = func(entryPath string) error {
// List directories
entries, err := storage.ListDir(volume, entryPath)
if err != nil {
return err
}
// Recurse and delete all other entries
for _, entry := range entries {
err = delFunc(pathJoin(entryPath, entry))
if err != nil {
return err
}
}
return nil
}
return delFunc(retainSlash(pathJoin(dirPath)))
}
Looking at the code above, one would assume that storage.ListDir (value receiver) is called immediately without any need for a pointer receiver version (cmd.(*retryStorage).ListDir) to be called in between. So what is going on here?
Autogenerated functions
Let’s first examine what this pointer receiver function is all about. By running go tool objdump -s ListDir minio we can get the definition of the function and see that it is apparently <autogenerated>:
TEXT minio/cmd.(*retryStorage).ListDir(SB) <autogenerated>
<autogenerated>:926 0x1f2e00 GS MOVQ GS:0x8a0, CX
<autogenerated>:926 0x1f2e09 CMPQ 0x10(CX), SP
<autogenerated>:926 0x1f2e0d JBE 0x1f2f42
<autogenerated>:926 0x1f2e13 SUBQ $0x78, SP
<autogenerated>:926 0x1f2e17 MOVQ BP, 0x70(SP)
<autogenerated>:926 0x1f2e1c LEAQ 0x70(SP), BP
<autogenerated>:926 0x1f2e21 MOVQ 0x20(CX), BX
<autogenerated>:926 0x1f2e25 TESTQ BX, BX
<autogenerated>:926 0x1f2e28 JE 0x1f2e3a
<autogenerated>:926 0x1f2e2a LEAQ 0x80(SP), DI
<autogenerated>:926 0x1f2e32 CMPQ DI, 0(BX)
<autogenerated>:926 0x1f2e35 JNE 0x1f2e3a
<autogenerated>:926 0x1f2e37 MOVQ SP, 0(BX)
<autogenerated>:926 0x1f2e3a MOVQ 0x80(SP), AX
<autogenerated>:926 0x1f2e42 TESTQ AX, AX
<autogenerated>:926 0x1f2e45 JE 0x1f2efa
<autogenerated>:926 0x1f2e4b MOVQ 0x80(SP), AX
<autogenerated>:926 0x1f2e53 MOVQ 0(AX), CX
<autogenerated>:926 0x1f2e56 MOVQ CX, 0(SP)
<autogenerated>:926 0x1f2e5a LEAQ 0x8(AX), SI
<autogenerated>:926 0x1f2e5e LEAQ 0x8(SP), DI
<autogenerated>:926 0x1f2e63 MOVQ BP, -0x10(SP)
<autogenerated>:926 0x1f2e68 LEAQ -0x10(SP), BP
<autogenerated>:926 0x1f2e6d CALL 0x5d5a4
<autogenerated>:926 0x1f2e72 MOVQ 0(BP), BP
<autogenerated>:926 0x1f2e76 MOVQ 0x88(SP), AX
<autogenerated>:926 0x1f2e7e MOVQ AX, 0x28(SP)
<autogenerated>:926 0x1f2e83 MOVQ 0x90(SP), AX
<autogenerated>:926 0x1f2e8b MOVQ AX, 0x30(SP)
<autogenerated>:926 0x1f2e90 MOVQ 0x98(SP), AX
<autogenerated>:926 0x1f2e98 MOVQ AX, 0x38(SP)
<autogenerated>:926 0x1f2e9d MOVQ 0xa0(SP), AX
<autogenerated>:926 0x1f2ea5 MOVQ AX, 0x40(SP)
<autogenerated>:926 0x1f2eaa CALL cmd.retryStorage.ListDir(SB)
<autogenerated>:926 0x1f2eaf MOVQ 0x48(SP), AX
<autogenerated>:926 0x1f2eb4 MOVQ 0x50(SP), CX
<autogenerated>:926 0x1f2eb9 MOVQ 0x58(SP), DX
<autogenerated>:926 0x1f2ebe MOVQ 0x60(SP), BX
<autogenerated>:926 0x1f2ec3 MOVQ 0x68(SP), SI
<autogenerated>:926 0x1f2ec8 MOVQ AX, 0xa8(SP)
<autogenerated>:926 0x1f2ed0 MOVQ CX, 0xb0(SP)
<autogenerated>:926 0x1f2ed8 MOVQ DX, 0xb8(SP)
<autogenerated>:926 0x1f2ee0 MOVQ BX, 0xc0(SP)
<autogenerated>:926 0x1f2ee8 MOVQ SI, 0xc8(SP)
<autogenerated>:926 0x1f2ef0 MOVQ 0x70(SP), BP
<autogenerated>:926 0x1f2ef5 ADDQ $0x78, SP
<autogenerated>:926 0x1f2ef9 RET
<autogenerated>:926 0x1f2efa LEAQ 0x96c2bb(IP), AX
<autogenerated>:926 0x1f2f01 MOVQ AX, 0(SP)
<autogenerated>:926 0x1f2f05 MOVQ $0x3, 0x8(SP)
<autogenerated>:926 0x1f2f0e LEAQ 0x976870(IP), AX
<autogenerated>:926 0x1f2f15 MOVQ AX, 0x10(SP)
<autogenerated>:926 0x1f2f1a MOVQ $0xc, 0x18(SP)
<autogenerated>:926 0x1f2f23 LEAQ 0x9707a2(IP), AX
<autogenerated>:926 0x1f2f2a MOVQ AX, 0x20(SP)
<autogenerated>:926 0x1f2f2f MOVQ $0x7, 0x28(SP)
<autogenerated>:926 0x1f2f38 CALL runtime.panicwrap(SB)
<autogenerated>:926 0x1f2f3d JMP 0x1f2e4b
<autogenerated>:926 0x1f2f42 CALL runtime.morestack_noctxt(SB)
<autogenerated>:926 0x1f2f47 JMP cmd.(*retryStorage).ListDir(SB)
So we can see that, about half way in at offset 0x1f2eaa, it calls retryStorage.ListDir (value receiver) which we have defined in our Golang code at minio/cmd/retry-storage.go:183. Before this it sets up all arguments on the stack which includes dereferencing *retryStorage and creating a copy of it on the stack (as retryStorage.ListDir needs).
After the CALL the return arguments are loaded (offsets 0x1f2eaf through to 0x1f2ec3) and copied into the appropriate slots (offsets 0x1f2ec8 through to 0x1f2ee8) for the return arguments of (*retryStorage).ListDir after which the autogenerated function returns.
So all that (*retryStorage).ListDir essentially does is wrapping the call to retryStorage.ListDir and making sure that the object pointed to by the pointer receiver is not modified.
Why does Golang do this?
Now that we know what (*retryStorage).ListDir does, the next question is why does Golang do this? This has to do with the fact that StorageAPI is an interface type, and therefore the storage argument in cleanupDir is an interface value which effectively is a two-pointer structure: Interface values are represented as a two-word pair giving a pointer to information about the type stored in the interface and a pointer to the associated data.
So when we make the call to storage.ListDir we are caught in the situation that we have access to a pointer to retryStorage but only a value receiver method for ListDir. This is then where the Golang compiler is nice enough to (auto)generate it for us, although it does come at a considerable execution cost since we need to dereference our object and copy the return arguments etc. Thanks to @thatcks for providing a clarification on this point.
What if you don’t want this?
Well, first of all you don’t necessarily need to ‘fix’ anything because there is nothing inherently wrong and your code will run just fine.
However if you do want to fix it, it is deceptively simple: simply change func (f retryStorage) ListDir(...) to func (f *retryStorage) ListDir(...) at minio/cmd/retry-storage.go:183 and you are all set as per:
TEXT github.com/minio/minio/cmd.(*retryStorage).ListDir(SB)
retry-storage.go:199 0x15edc0 GS MOVQ GS:0x8a0, CX
retry-storage.go:199 0x15edc9 CMPQ 0x10(CX), SP
retry-storage.go:199 0x15edcd JBE 0x15efe5
<rest omitted>
Obviously you will want to make sure that your implementation of the function does not by accident change the value of f because you will then start noticing this (which would most likely be a little weird anyway because presumably the intent of changing the contents of f would be of the caller to take notice of this which does not happen with value receiver functions).
And, as a free bonus, you will shave a couple of hundreds of bytes of the size of your executable (for just this single function, not a bad result for just adding a single * in your source code file…).
Conclusion
Hopefully this blog post will have given you some insights into how Golang functions internally and clarifies what these autogenerated functions are about and what you may do about them.
In a later post in this series we will also elaborate more on the performance of pointer receiver versus value receiver functions.