We review the different data locations supported by Solidity and the implications of using each of them.
This post is the second in a series of three on solc. In the first, we covered how Solidity passes arguments to functions via the stack. In this post, we cover the different data locations supported by Solidity: When can one use them? What are the caveats or advantages of using each data location in terms of code complexity (and gas usage) as well as the persistence of the data?
The EVM stack is great for holding values of size < 32 bytes, but what if one tries to pass non-value type variables, e.g., dynamically sized variables like string or bytes? We can encode these in the calldata to an external or public function, but how are they passed to the callee? The EVM stack can’t really handle them… In this post, we’ll learn exactly how. Here’s the conclusion, upfront:
Note: we’ve briefly covered some of the assignment rules that apply to each of the data locations, but readers may also want to familiarize themselves with the official Solidity documentation section on this here
Below we’ll go over the 3 data locations in-depth, provide examples and explain the differences in implementation as well as how choosing each of those, changes the outputted EVM byte-code.
memory
As the name suggests, this data location stores the variable in the EVM memory.
To explore the memory data location, let’s first go over the basics of EVM memory. Memory in Solidity is linear. Each byte in memory can be directly accessed by an index. Each word in memory is 32 contiguous bytes, the same size as a storage slot, and words are laid out in a straight line, one after another. Solidity reserves four 32-byte slots for internal use:
The free memory pointer points to 0x80 initially, and this can be seen at the very beginning of the runtime code for all modern solc compiled contracts:
// bytecode
0x6080604052
// assembly
0x0: PUSH1 0x80
0x2: PUSH1 0x40
0x4: MSTORE0x40-0x5f reads 0x80 after the MSTORE operation. This sequence of bytes, being unique to solc-compiled contracts, is often used by bytecode analyzers to detect the compiler type.
Solidity places new objects stored in memory at the free memory pointer. Memory is not freed throughout the transaction — the free memory pointer simply points to the next chunk of free memory and memory grows with each new allocation.
Using the memory data location is allowed within function argument assignments, i.e., in the function declaration, as well as inside the function itself:
Variables located in memory are only accessible (via Solidity) from within the scope of the function in which they’re defined.
Remember that memory is never freed, hence the variable can be accessed (i.e., via inline assembly) by simply reading from the memory location at which it was allocated.
These are the important rules to know about the memory data location, but we still didn’t answer how s1 actually gets passed to the doStuff function. For that we should try to compile an extremely simple contract:
As can be seen above, the contract only has one function that receives a string tagged with the memory data location and does… nothing!
The compiled version should have a nice, simple control-flow graph (CFG), right?
We’re not going to try decipher what’s going on there… It doesn’t really look like it does nothing 😕.
Luckily there’s a better way; we can compile the Solidity code to its Intermediate Representation (IR) for more readable Yul expressions:
~/.solc-select/artifacts/solc-0.8.13 --combined-json bin-runtime --ir ~/Desktop/stuffDoer2.solAnd this outputs the following IR for us (we’ve redacted some of the output for brevity and kept only the good stuff):
That’s much better. We can now skip straight to the external_fun_doStuff_7 function which is called from the dispatcher.
Right after enforcing a non-payable execution, we see the call to abi_decode_tuple_t_string_memory_ptr:
This is the function that does all of the heavy lifting that we saw taking place in the CFG image. It loads the string from the calldata to the memory and passes a pointer to that position in memory to the actual function that needs to be called: fun_doStuff_7.
The calldata is checked to be larger than 32 bytes in size, which is obviously mandatory if the ABI-encoded string is indeed valid. We also see the calldataload opcode, which loads the first 32-bytes of the ABI-encoded string and makes sure it’s not greater than the maximum size allowed to be loaded to memory (0xffffffffffffffff).
Why would solc impose such a limitation on data being copied to memory? Folks who read the Yellow Paper (anyone? 👀) might remember that it explicitly states that the memory contents of the EVM is just a series of 0s of size 2**256:
So how come the limitation in solc is only 2**64?
While the official reasons to that choice are unclear, there are a couple of hints we can use to find out why this value is the chosen limit. If we take a look at the most popular client implementation (geth), we can clearly see that they chose to use Uint64 to represent a memory address:
Moreover, this decision sort of makes sense when we zoom out a little bit and think about what it means to actually use very high addresses of memory.
The above statement from the Yellow Paper states casually that the memory contents are continuous from position 0. This actually has implications as it means that every time an opcode accesses some memory address, it expands the memory such that if the memory bounds were previously:
[0, … , mem_pos]
The new bounds of the memory would be:
[0, … , new_mem_pos]
There is also a gas cost that the user pays for each additional word (32-bytes) when expanding the memory. This disincentivizes accessing higher addresses of memory as the cost would be enormous. Remember, you pay for every word between the current mem_pos and the new_mem_pos; the higher new_mem_pos is, the more gas you pay.
We urge readers to check out the math that is involved in calculating the costs for memory expansion here.
Back to our little contract that “does nothing.” After checking the offset of the string to be copied to memory, we are now ready to jump to another function abi_decode_t_string_memory_ptr:
Here we can see the length of the string being loaded from the calldata. Remember, the first word read from the offset is the data length in the case of an ABI-encoded string or bytes.
Then we see 0x20 added to the offset, to denote the point at which the actual data of the string starts, and it is passed to yet another helper function abi_decode_available_length_t_string_memory_ptr:
In the above block we see all of the inner helper functions that together finally allocate the string in memory. Once again, we see the length check we saw before, only that now it happens as part of array_allocation_size_t_string_memory_ptr.
You might wonder why perform the same length checks over and over again. The reason is that these helper Yul functions are actually generated from templates (written in a template dialect called Whiskers) and are sometimes added to the generated code without the necessary context to know that some check might be redundant.
For instance, the template that generates array_allocation_size_t_string_memory_ptr can be seen here:
Even the function name is templated. A different function is generated for each type to be decoded. Also, the template language supports conditions on various properties of the arguments passed to the function creator. Pretty neat!
Looking at array_allocation_size_t_string_memory_ptr, we see that the byteArray condition is resolved to be true. This means that the size of the string, which is treated the same as bytes, is rounded up to a multiple of 32 . The length of the size slot, 0x20, is added to it.
The actual allocation is allocate_memory . It triggers a series of memory allocation helper functions:
allocate_unboundedgets the free memory pointer.finalize_allocationperforms the following steps:
- Computes the new free pointer by adding the size calculated in thearray_allocation_size_t_string_memory_ptrfunction to the free pointer.
- Checks that it won’t overflow the memory(< 2**64).
- Stores the address of the new free memory pointer in the
memory pointer address (0x40).
Then, back to abi_decode_available_length_t_string_memory_ptr, the length of the string is now MSTORE’d at the beginning of the memory pointer.
The actual length of the string as reported in the ABI-encoded calldata is now checked to be valid, i.e., not exceeding the actual end of the string, and then finally a call to copy_calldata_to_memory occurs:
This copies the calldata to the destination in memory whose address we computed earlier and clears the following 32 bytes by setting them to 0.
At the end we get returned a pointer to where the string starts in memory and it is passed to fun_doStuff_7, just like we expected.
To recap what we observed when a dynamic parameter is passed to a function using the memory data location:
- The parameter is decoded from the
calldata. - The parameter size is checked to be <
2**64. - The parameter is loaded into the memory starting at the next free memory pointer address.
- A pointer to the parameter’s start offset in memory is then passed on the stack to the function.
All of the above steps are executed by the callee, as can be seen in the logic of external_fun_doStuff_7.
calldata
This data location passes the variable to a function via the calldata. It’s only available for arguments passed to external functions. Variables marked with calldata are immutable and cannot be modified. Like with memory, they live only in the scope of the callee and are non-persistent. Let’s observe the differences between memory and calldata by taking a look at a simple example:
The CFG:
Compiled with the helpful — ir flag, we get:
Just from the first impression of the generated Yul code (or the CFG, if you generate it), we can already see that passing arguments in the calldata to a function is much simpler than memory.
abi_decode_tuple_t_string_calldata_ptr receives the start of the data part of the calldata, after the 4-byte-long sighash, and the end position of the data, which is the end of the calldata in our case.
The offset in the data at which the string starts is then read from the calldata; we see the usual 2**64 check, then a call to abi_decode_t_string_calldata_ptr, which simply returns the string’s position in the calldata and its length.
Remember, variables located in calldata are immutable, so that call is all we need to read the string. As we saw earlier, variables in memory can be modified and are therefore stored in storage. This is why calldata is the cheaper way of passing a variable if it only needs to be read by the function.
storage
This data location passes the variables to a function via contract storage, which is, of course, persistent. This data location can only be used in private/internal functions. Let’s explore with another simple example:
This is quite a simple contract that helps us demonstrate how values are passed from the storage to a function.
The CFG is also quite friendly:
And the Yul IR:
We’ve re-arranged some of the functions that we’re interested in so that they appear together.
In fun_doStuff_12, we see the pointer to the storage slot where our item to pass is located is passed to fun__doStuff_22. This is simply the address of the storage slot, 0x00 in our case.
We then see the callee preparing the grounds for the value to be returned from memory. Yes, the memory data location can be used to return values too! This happens in the zero_value_for_split_t_string_memory_ptr , which returns a pointer in memory that will hold the value to be returned.
Our storage slot pointer is then passed to convert_array_t_string_storage_ptr_to_t_string_memory_ptr. It loads the value to a memory item and passes the pointer to it. It does so by calling copy_array_from_storage_to_memory_t_string_storage_ptr. This function allocates memory as usual, using the allocate_unbounded function we already know, then calls abi_encodeUpdatedPos_t_string_storage_ptr_to_t_string_memory_ptr . It, in turn, calls abi_encode_t_string_storage_ptr_to_t_string_memory_ptr, which does the heavy lifting:
We finally see the actual contents of the storage slot being loaded to the stack using sload. Then we see the length of the string inside the storage being extracted using extract_byte_array_length:
Here we see the check for the lowest order byte in the storage slot’s contents. If it is set, that indicates we’re dealing with out-of-place data, i.e., a string longer than 31 bytes and thus stored outside of this storage slot. We will soon discover exactly where it is stored. The length of such a string is extracted by dividing the value of the last byte in the storage slot by two and then subtracting 1 from the result. This subtraction occurs automatically as the EVM DIV operation “floors” the result.
Otherwise, we have regular in-place data that is stored in the storage slot we just read, meaning the string is at most31 bytes. The size is just the last byte divided by 2.
We then see the call to array_storeLengthForEncoding_t_string_memory_ptr, which stores the length of the string in memory and updates the position of the memory pointer:
We now see the switch-case statement that actually copies the string from the storage to the memory location we allocated before:
We see the test of the last byte: if it’s not set, then a short string (31 bytes long, due to the and(slotValue, not(0xff))) is stored in memory.
Otherwise, a for loop actually reads the string 32 bytes at a time from the real position of the string data in storage that is determined by array_dataslot_t_string_storage_ptr:
The data of the string starts from the storage slot in keccak256(<storage_slot>). The memory pointer is then returned from the function as expected.
Just like we saw with the memory data location, the callee is responsible for executing the logic that copies values passed from storage.
Thank you to Yoav Weiss and the smlXL team members who offered advice and feedback for this post.
Respond to this post if you have any questions, and Tweet at us if you have a topic you’d like us to cover.
We’re hiring!
