Technical discussion on the development path for Ethereum light clients, enabling resource-constrained devices to interact with the Ethereum network without running a full node.

Transcript

[00:00] SPEAKER_00: Is the Ethereum light client protocol. So just to give an introduction to why clients are needed, I guess, the important thing here is that we are dealing with distributed consensus and we're dealing with a system where every full node has to process every single transaction. So right now with something like Bitcoin it's relatively fine. One transaction a second, anyone can do it. Although you do need to download about 20 gigabytes of stuff by now, which is a problem.

Now with something like Ethereum, in some ways our protocol is more efficient, in some ways it's not sufficient because we have to store a bunch more stuff. But if scalability gets, if usage gets to any significant points, if you imagine we have something like a thousand transactions a second, then the amount of resources that it will take to run a full node on the network is 100 to 200 megabytes per hour just processing transactions. Assuming 10% of those transactions increase the state, which is about accurate. For Bitcoin, that's a gigabyte of hard disk space every two or four days. Thousand gas a transaction, that's a million gas a second. I believe that's pretty much 100% of all the Go Ethereum can do right now, if not more.

[01:25] SPEAKER_01: Yeah, we will have just inside compilation at that point of course.

[01:29] SPEAKER_00: Sure.

[01:34] SPEAKER_01: So ultimately it's kind of important that for decentralized applications we have a strategy for using these things on mobile devices. The general idea is to replace so many massively multi-user applications, the things we tend to use the web for, that these devices are going to be super important for actually providing people with what they want.

So we have two problems. The first is given that a light client knows very little about the blockchain because it stores very little, probably only the headers, how does it find out about events that are going on in the blockchain that the decentralized application might be interested in? The second is assuming that it now knows about these events, perhaps it's a transaction, perhaps it's a particular execution and a particular contract, how does it be certain that this information is actually true?

[02:33] SPEAKER_00: So just to give an introduction to the fundamental theory behind light clients, everything is based on this wonderful construction invented by Ralph Merkle in the 1970s. Merkle trees. So the idea behind a Merkle tree is you arrange data in a kind of tree structure where you have all the little nodes, keys and values at the bottom and then you take a few nodes, make a hash of them. Then you take, and then that's the next level, and then you take a few hashes and make a hash of those and it all sort of bubbles up until we have a single root hash at the top. And you can think of the root hash as kind of representing the entire tree. So there's no way to change a single, anything whatsoever about the tree without creating changes that eventually propagate up the tree and ultimately lead to a change in the root. So if the root is inside the header, then once you've validated the header, you've sort of implicitly validated the entire tree.

So the idea with light clients is you only download the block header chain by default. And if you need data, then you just fetch whatever data you want. And once you have that specific set of data, you can just validate it by just making sure that the hashes match all the way up. So the basic idea is that you can think of a proof of something as being the subset of tree nodes that are involved in accessing or processing it. So if you want a proof that some particular account has some particular state, then you take the nodes in the tree that you need to sort of go all the way down to that particular thing, and that set of nodes basically is the proof.

So the way you create a proof is literally this, but this is how it's actually implemented in the Python client right now. If you want to prove some particular part of an account, you just grab it, but you record all the tree nodes that you accessed. And the way you verify the proof is that you just run the same standard algorithm, but using the proof as your database. So the nice thing is that with the way the protocol is designed, a lot of this stuff you just sort of get automatically because everything is based on these reverse hash lookups. And if you assume that the hashes that you need actually already exist, then you could just reuse the exact same code. It's just a matter of which hashes do you need or which little nodes do you need to download to make sure you have everything?

So in Ethereum we have three Merkle trees. We have a state tree, state tree records account balances, storage, a little Merkle tree for each account code. In 1.1 maybe we'll consider events. So a way for contracts to sort of do events that only get propagated, processed in the future and then those events have to be stored in the state in the meantime. Transaction tree just stores all the transactions in a block. So the receipt tree, that's pretty important and we'll talk more about that later. But the idea is that it sort of stores specific results that are related to processing each individual transaction.

So what do we care about proving or what do we care about light nodes being able to securely access? Account balance, account nonce, account code, account storage data, whether or not a transaction has been included. So one thing I know, I was asked a couple days ago is how does a node even, or how does the JavaScript even know that a particular transaction was processed so they can start sending off another transaction? And this is something that the light client should be able to figure out from the blockchain. Also potentially if your light client wants to start actually probabilistically validating that a block was produced correctly, they can do that. And finally a thing that Gav will probably talk more about, which is logs.

So chain data, we already talked about that prove existence. So if you want to fetch a chain data you just sort of prove an existence of a node in the tree. So transaction state transition validity is a more complicated thing. But basically if you want to prove that a particular transaction was processed correctly, then you take some of this data from the receipt tree. So the receipt tree contains an intermediate state root. After processing every transaction take the intermediate state root, you take the transaction and you make sure that if you just run the transaction on top of the previous intermediate state root, you get back the new intermediate state root.

And why is this stuff necessary? Basically one reason is that if you want a somewhat stronger assurance that things are being done correctly and if you don't necessarily trust the majority hash power. Once proof of stake comes along and you can't just rely on difficulty as a sort of proxy for security, it might become even more important. And finally Alex, so.

[07:23] SPEAKER_01: One of the things that we want to provide with dapps is the ability to effectively not think about the blockchain, not have to think necessarily about transactions and not have to think particularly about state. The idea is to provide an abstraction mechanism for the blockchain, for what the blockchain provides. And this abstraction mechanism, the sort of interface it provides, becomes something that we're a bit more familiar with. The notion of a sort of virtual machine.

As the virtual machine executes, it's useful because it's not in an environment that you have direct access to. You have to do it in terms of transactions. It's useful to be able to put things like checkpoints in to see if a particular contract that you're interested in is at a particular point in its execution. Such a point for example might be that it has received a payment, that a set of payment has been sent from one of your accounts, or that one of the orders that you might have put on an exchange for instance has actually executed. So this is kind of like the IO, more the O from the blockchain and the I into the rest of the dapp, but the IO between blockchain and the dapp as a whole, the JavaScript portion that's actually executing locally in the browser.

Now originally we had this idea of messages. So like transactions, messages are things that can actually go and execute contracts. But messages can also be the things that messages execute. Transactions of course always come from an external entity and they're cryptographically signed. And we had the idea of message tracking, message watching, whereby the JavaScript, the local portion of the dapp could watch out for changes on the blockchain that involved a particular message and it might also have filtered it by what the input to the message was.

Now this turned out to be less than ideal and so we sort of evolved this notion of a log. Now the log is like a breakpoint or a record. It allows a message, a contract to point out to an interface that a particular thing has happened. Logs can include a topic or many topics and these sort of identify the type of entry. Logs also implicitly include the address of the contract that created the log. And finally logs can also include data, just arbitrary unformatted binary data. And of course this data can mean whatever the contract wants it to mean.

The difference between the topics and the data is that the topics are indexed. So topics can be watched out for by the JavaScript front end, by the rest of the dapp. The reason that we have logs at all is so that we can very efficiently index them and filter them when dapps require that. Now because the light client is an important part of the eventual protocol, it's important to get this interface in now so we can work out exactly how light clients will be able to work out efficiently whether any of these log entries in blockchain have actually fired, whether they're there, whether new records have come along that we're interested in. And the way that we do this, as Vitalik mentioned before, is transaction receipts.

[10:42] SPEAKER_00: So first of all, just to describe how logs are implemented, there is an opcode. And this sort of bitcoiny term for what the log opcode does is proof of publication. So it just proves that something, it sort of publishes an event in the blockchain by just connecting it with a hash tree to a state to a written block header. Log opcode has up to six arguments. So first of all it has two arguments to grab a slice of memory that it keeps as data. Then a log can have up to four topics and implicitly a log always has as a sort of argument the address of the contract that created the log and the data structure is that, how a log is stored is this RLP: address, topics and data.

So receipts in the transaction. So the way that the transaction receipt looks is that this med state is like the state root after processing that particular transaction. Gas used is a counter of how much gas was used after processing the transaction. Bloom filter, we'll talk about that a bit later. And logs. So that's just a list, the RLP encoded list of each log where each log is encoded like this.

And in the block header we have the end state to match the intermediate states, gas used at the end. And we have a sort of block-wide bloom filter. So each bloom filter is 64 bytes. And the thing, so the way bloom filters work, just if you're not clear, it's a sort of mechanism where you can put keys into the bloom filter and the bloom filter is this really compact structure where you can then check other values against it. Once you've put a subset of keys into the bloom filter you can check some new value against the bloom filter and the bloom filter either returns no, this key is definitely not in the bloom filter, or maybe this key maybe is in the bloom filter. And the reason why it's maybe is because it's a sort of lossy compression where all the keys are in a sense hashed and packed together, so you're losing some data.

So the point is that the way the protocol works is you search the blockchain for bloom filters that are matching some particular topic that you're sort of watching for. If you find a block that matches what you want, you check every transaction. You see which transaction or which transactions match that have a bloom filter that matches that particular topic. And then once you find transactions, you would actually ask for the Merkle proof going down to the logs, which would tell you whether or not logs were actually there and if logs were there, what the actual content of the logs is.

[13:17] SPEAKER_01: So the minor thing that has been missed from this particular list is that we can search the header for blooms that tell us that a particular block may contain a transaction that had a log that we're interested in following. Then we can actually request the transaction receipt block and search in that block for the bloom filters. So the blooms are also for each transaction actually put in that block before we actually go ahead and search for the log entries themselves. So there's a little more efficient before.

[13:48] SPEAKER_00: Well, search blockchain for blocks, search block for transaction, search transaction. Well, I guess, well, not too clear, but. Oh, well.

So yeah, so basic roadmap is that the client that we're going to have for 1.0 is archive node. Simplest thing to implement. It stores everything. Then after that, what we want to get into is we want to move from archive node to what we're calling full node. So full node stores the entire state, but they only store recent history. So you can imagine they store the state roots going back maybe a week, and then they just store some more block headers going back maybe a month or a month or two, and transactions going back that far as well. And then if they want to grab ancient history, then they would just maintain some block headers, and they would use basically the light node protocol to ask for ancient history. And so full nodes are actually much closer to light nodes than to archive nodes in some sense. And then light nodes will just download block headers by default. And we are expecting it will take quite a while for the full sort of light line technology to get rolled out, just because it involves a whole bunch of these various prerequisites.

[14:57] SPEAKER_01: Yeah.

[14:58] SPEAKER_00: So from a. Right. So how does this work from a networking standpoint?

[15:02] SPEAKER_01: So what's important is that when a light client wants to operate, it does not have to request a proof, which are necessarily things that are customized for the requester from some other node. If that were the case then effectively light clients would be literally clients and the nodes that they come from, the archive nodes or maybe the full nodes would be servers. And that's not an especially nice way of doing it. It sort of reduces the decentrality of the network.

So rather than, so as an alternative to the ask for the proof mechanism, which may still be necessary for some particular things depending on whether the state tree is well distributed and well stored by the network, is to store the various portions of things like headers, blocks and transaction receipts on the DHT itself. So these are naturally things that are identified by the hash. They're actually identified by probably the same hash within the blockchain itself. And the idea is that light clients themselves are a contributor to this DHT. So they do their own bit in terms of storing maybe some transaction receipts, some blocks and some headers, that they can obviously should the chance happen that they need it, they can use themselves but that they can in a BitTorrent-like tit for tat fashion exchange with other light clients in order to provide information across the network without having this reliance upon archive or maybe full nodes.

[16:42] SPEAKER_00: So yeah, so the idea is that once again it actually sort of works nicely from an implementation standpoint is that the way that it's, all you're doing is that you're grabbing, instead of grabbing nodes from a database, you're just grabbing nodes from this DHT. So you're just sort of using the, you can think of it as like using the Internet as a sort of extra hard drive. And this can be used for reading the state, it can be used for processing watches, potentially probabilistic validation.

So as far as what we want to have is we want to store everything that light nodes are going to be storing by themselves in this kind of specialized DHT. We don't necessarily want to mix it with Swarm just because it's better to have, first of all it's better to have the different protocols be separate. I think people should be able to use Ethereum even if they don't care about Whisper and Swarm. They should be able to use Swarm if they don't care about Whisper and so forth. And there's a possibility of making a sort of tit for tat protocol for where light nodes can sort of ask each other, give each other proofs and that helps incentivize nodes to actually store things.

So long term, light clients and proof of stake. It's actually, there are potential tricks that could, it's obviously going to have more info load than proof of work because with proof of stake you actually need to validate that people have stake. There's this option of sort of artificially discretizing the protocol. So you would have these sort of checkpoints that are far away from each other and each checkpoint, and the idea is that each checkpoint would be signed by a very large number of nodes and then you would see that, hey, this very, very large set of signers has validated this next checkpoint and then you sort of go from there.

Proof provision, not incentivized. I mean tit for tat protocol sort of does it. Putting it on a DHT sort of solves the problem. Is there some better solution? Will we need to have some better solution? Hard to say. Light clients in Ethereum 2.0 depends entirely on what Ethereum 2.0 is going to look like. Anything else you want to add?

[18:44] SPEAKER_01: No, that's about it.

[18:45] SPEAKER_00: All right, that's about it.