Web3 Galaxy Brain 🌌🧠

Nicholas: Welcome to Web3 Galaxy Brain. My name is Nicholas. Each week, I sit down with some of the brightest people building Web3 to talk about what they're working on right now. My guest today is Ryan Smith, founder of Index Supply. Index Supply is a company dedicated to open source indexing the EVM. Index Supply builds on Ryan's years of experience building backends and indexing services at Heroku, Chain, and Mint.Fun, and through consulting with Zora, Reservoir, and More. On this episode, we discuss Shovel, Index Supply's open source indexer that transcribes every event emitted in an EVM node to an OLTP Postgres database. We discuss the ins and outs of indexing, Index Supply's companion block API, and the company's unconventional approach to funding open source software development for long-term sustainability. It was a pleasure catching up with Ryan, who is generous with his knowledge and opinionated in his indexing. I hope you enjoy the show. As always, this show is provided as entertainment and does not constitute. legal, financial, or tax advice or any form of endorsement or suggestion. Crypto has risks, and you alone are responsible for doing your research and making your own decisions.

Ryan Smith: The most fun thing to do after college is creating your own instrument. a startup a small company of about 15 people uh then called heroku um and yeah and i cut my teeth and um and and learned a lot about uh building at the the lower levels of uh of the stack. um but yeah yeah i started out at heroku 2007.

Nicholas: it started i think so it's. is it fair to say that it's something like that era and maybe the first real like what today you might think of as vercel?

Ryan Smith: oh for sure and in fact i would. i would claim that the platform as a service like that concept uh was pioneered by heroku. i i think um if you kind of rewind time uh you know 2007 to 2012 kind of that era. ruby on rails was the um the really hot technology and it was people. it was and and honestly like people might remember the blog in 15 minutes that the the infamous dhh video where he demoed ruby on rails and how to build a blog in 15 minutes. that was groundbreaking uh and i don't think anyone had ever. it was just like a. it was like a net new thing to like the web developer space and the web itself was you know just kind of getting its foothold i would say um and so at the time the the way that you it was. it was wild. how much work and how much effort it took. and back then even amazon wasn't reliable like you had to do all of these like an ec2 instance. um today it's kind of like a thing that you just take for granted like it's always there very seldom restarts like there's always more if you need them um but this wasn't the case back then. and and and and and so that's like the high watermark so like the long tail like oh man there's linode there's dream. it was a good thing called dream host. there were like a few hosting companies that people would use and it would just. it was a tremendous amount of effort to get a web app deployed onto the internet. so heroku and i remember i was in college doing um i studied math but i was doing uh web dev to kind of pay my way through college and i saw heroku and it was just like a. it was a mind blowing moment where i was just like oh this is the coolest thing ever i i have to work there. so i just kind of made it my mission after i finished college to go get a job there.

Nicholas: that's great. that's ended up becoming kind of like. it became very popular in the boot camps. right it became a sort of fast way to become a web dev. or is that a false impression?

Ryan Smith: no no totally um and you know boy i don't know. i mean even today i still i don't use ruby that much these days. i haven't touched ruby in a long time. um that being said i know that you can get a lot done in a very short amount of time. um with with with ruby on rails it's not as hot it's not as trendy um you know as some other technologies to use but i still think that it it has a place. um you know for certain situations i guess it really depends on the person. honestly um but people i i let me put it this way i still know people today who are building new projects using ruby on rails which is just fascinating

Nicholas: right i like how uh i think it's levels ios take on this which is kind of like it's just whatever tools you're good with are the right tools to use and it definitely feels in the javascript ecosystem like there is this kind. i mean some people will claim it's i don't know uh fang engineers justifying their own salaries by constantly changing the packages that you're expected to use this kind of fashion driven uh npm development. but regardless of what the the motivation is uh it does seem like there's. maybe i guess it's just easier to talk about uh tooling and changing around your tooling and improving the performance of your tooling rather than building a product people actually like to use.

Ryan Smith: yeah yeah and there's like a very human thing too i i know even in like the ethereum space like the the language wars and like the technology choices um it becomes a very religious topic very quick and and i think it's just a thing that humans like you know intertwine themselves with like these um tribes or these uh groups.

Nicholas: you know definitely this tribalism i remember. i mean it feels like ages since we've had people like arguing about windows versus mac versus linux etc. but that just took so much attention for so many years people worrying about this problem. that's like i don't know. it's. it's a different. they're different tools. why why bother so much? you also worked on uh chain. right it was chain like um. i heard you describe it as like a bitcoin api or like a blockchain api. is that? is that really what it was?

Ryan Smith: so the company took a few different forms. um we started off uh as a bitcoin api so you could think of it as like um infura or alchemy but for bitcoin and we had started the company in 2013 and so this was before ethereum had launched. the yellow paper was uh you know circulating and and and and and and and and and back then you know my exposure to vitalik for example was i just knew him as like the really smart young fella who wrote for the bitcoin magazine and had like all these like clever ideas for like what we could do with bitcoin um and then when the ethereum project really started to take hold you know it definitely was captivating and you know embarrass embarrassingly i personally dismissed it. for many years i um yeah and and i would even say that i you know for a long time i had this really weird friction i would say with the blockchain space um and i and to be honest a lot of my colleagues you know that i i knew from my heroku days and just from my time in silicon valley. a lot of my colleagues are still there and i think you know there's this initial like. what is this technology like it? it didn't come from the traditional you know places. it um seems scammy you know and so people can just very quickly write it off. and i think even though i was like starting a company in the space and i had worked in the space i had always kind of seen crypto as like this. it's like oh man it's like kind of edgy like i kind of wish they would just have gotten in line with like the traditional silicon valley you know like mode of operation and and so i had kind of developed like uh yeah like a little bit of friction uh to like the full decentralization. you know the the galaxy brain idea of crypto. i guess you could say i i had developed a little bit of dissonance with that and you can see that through what we did at chain. um so you know we started off as a bitcoin api where we were running bitcoin nodes and we were building you know like a push notification system on top of it and we were doing some clever stuff with off return. like there's this cool thing that you can do with the scripting language where you know you can sneak some data in and people are doing colored coins and things like.

Nicholas: that is using a similar thing these days.

Ryan Smith: yeah they are yeah uh not exactly off return but like you know similar in spirit.

Nicholas: yeah um just to be clear for the listeners you were co-founder and cto at chain uh from 2014 to 2018. yep

Ryan Smith: exactly yeah and so and so you know once once we kind of realized that like oh bitcoin isn't really the the platform that people are going to be developing applications on we pivoted um and and this was like in the 2014 2015 era there was like one of the memes during that time was uh blockchain not bitcoin and there was just tremendous amount of i would say like incumbent interest in the technology of bitcoin and everyone wanted to have a blockchain and enterprise. blockchain became a thing and so we had moved the company into um into a place where we were building our own blockchain and we had put together a really amazing team. um a lot of people have since moved on to like amazing projects in the space like um. henry de valence of penumbra was on the chain team. dan robinson who's now at paradigm was on the chain team oleg andreev. um who's uh doing tawn? i was on the chain team test at ok lab. so we had really built like an amazing team and we were working on a a private blockchain. so we had developed our own um kind of federated consensus model our own our own virtual machine our own scripting language. um henry was working on all this like crazy. um amazing uh privacy technology and we had a lot of um. so there was a lot of really cool stuff happening there. but um i think at the um i think as we've seen the incumbents just like aren't they're they're not gonna make it. and and i think my personal tune kind of changed. um you know after leaving chain and after just taking a breath and looking at the space i got decentralization pilled. i would call it where i just like i worked on several big projects at like city and nasdaq and visa and i got so burnt out about it working in that space that i decided that you know for the short amount of time that i have on this earth i want to be spending you know my personal energy working on a whole new system. like i don't want to incrementally change the existing stack i would like to put on the full decentralization. you know i want to go like full-on wizard. you know like i want to work on the stuff that seems stupid but if it works it would be really cool. that's that that's kind of where i wanted and so that's how i ended up you know doing stuff at index supply. was i kind of morphed into this the space of like? no i actually just want to position myself to be like full debt on decentralization and it's going to take. you know i have a longer time horizon than i did with chain like. i'm thinking you know 10 20 20 30 years out and that's and i want to be putting my energy kind of towards along that arc if that makes sense totally so so basically you're.

Nicholas: you're thinking is that indexing of public blockchains is just going to be more and more important so you can set yourself up for a long time horizon and be directionally correct even if day-to-day you're not watching the the the charts for the the the minute trading uh uh movements or anything like that precisely.

Ryan Smith: and i think that like if if this technology works at all and to be determined if it does you know it's like there's a certain lindy effect that we've achieved like up until this point. but you know still the jury's out. you know whether we actually you know rebuild civilization on this new tech stack. right but if we do uh then that i want to be part of that and the way that i see myself being part of that is i'm just like combining my interest of um basically system administration and like low-level back-end move you know move bytes from machine a to machine b like this is the thing that since i was a kid um is this more back-end kind of computer work. and then i combine that with like you know ethereum the the future you know tech stack of civilization. and so that's kind of where i'm positioning myself. and index supply is it's like okay people are going to need to have these these good unix like tools um to interact with ethereum. and basically you know if if civilization does move on to this new tech stack then there's going to be you know there's going to be a huge market of people who need to interact with with these systems and so there's going to be plenty of opportunity for different companies to have you know different types of tools to interact with this new tech stack. um.

Nicholas: so so that's kind of where index supply sits and you've uh had beyond your background building infrastructure and and chain and the kind of blockchain api world and moving into private blockchains. you've also had more recently experienced building indexers for projects that are active on ethereum and on other evms. i'm thinking of a lot of people who are kind of like uh mint.fun. uh you had a part in building their indexing system. maybe you could explain a little bit about uh mint.fun's indexing needs and maybe how they relate to other app dev shops and and really what what is the need for an indexer in the first place?

Ryan Smith: totally totally yeah. so yeah. so i uh i had the opportunity to work with um with with luke and the wonderful team at at a mint.fun um. and when i had started working with them um honestly just before we had launched mint.fun. before that there was an app called context and which is the company name and uh it was um it was kind of um. i almost got social graph overlaid on top of um various nft projects um and then the company later then pivoted and put all their effort into mint.fun. but um before that pivot i was working on building an indexer system for them. and uh they were. they were on top of the graph and um and then they were using some hosted nodes and and this was before the l2 revolution. so it was really just you know mainnet and gorley. and so uh yeah and and i looked at the situation um just kind of like with a fresh perspective and i was like boy like i feel like we could just run a bit you know run an ethereum node uh on an ec2 instance and and all the data is would would then be ours on a hard drive that we pay you know a few hundred dollars more. and uh and then we could you know we could read off that hard drive as much as we want and as fast as we can. and uh and so that means that we could index things really quickly and we could index and re-index and re-index as many times as we want and we just paid our flat monthly fee. so it so for my perspective

Nicholas: model where you're paying per request

Ryan Smith: exactly yep yeah so and

Nicholas: this is like particularly pertinent for context or for mint.fun where i mean context was like following wallets and then getting a kind of og facebook style feed of things that those wallets have done or trending collections amongst the people you're following and then i guess some similar kind of database manipulations in order to generate the mint.fun uh which for people who haven't seen it is like top trending collections based on how many mints they've had in the last hour 12 hours something like that.

Ryan Smith: yeah yep yeah great great description um and and so i i guess the opinionated approach that i brought to solving you know the the tech problems of building a site like mint.fun was i thought hey inside the ethereum full node there is uh you know all of the receipt and log data in its various encoded formats lying on the hard drive. and so if we can just build a system that um knows how to like scan that hard drive quickly and it knows how to decode log data that's on there and if we can then copy that data uh and just be very efficient with the copies if we can just like bring over the minimal amount of data that we need and if we can copy that into a postgres database uh then we can um write all these interesting queries uh on top of that postgres database uh to produce trending. you know nft mints that are happening in real time um so you know if you wanted to do that sort of thing it would be very difficult uh to do with just an ethereum node and the standard rpc that it offers. um it would be wildly inefficient and it just it's just not the right abstraction because

Nicholas: you have to be doing things like multiple queries or batching queries in order to make comparisons in order to come up with the list for example of the mint.fun top trending collections totally like you'd basically be writing a mini indexer inside of each request in order to basically you need some kind of database in order to have have that kind of interface.

Ryan Smith: yes exactly you you want to put you know you want to organize the raw data in different ways so that you can query on it uh more efficiently which i guess you would call an index. so this is kind of why people call it like indexing the blockchain. it's like it's like. well yes all the data is in an ethereum node. but if for certain types of queries um you want to make the queries fast and so to do this um you should create you know you should create some alternate forms of that data that are designed for querying which is commonly known as an index.

Nicholas: well i mean you've been working on this indexing problem for so many years. i'm curious like are there trends in the theory of indexing blockchains? like? i don't know even enshrining some kind of database that is more useful than just the raw blockchain data directly into the blockchain's design or tendencies in different types? i mean you're you're saying postgres. maybe have there been other approaches to creating these kinds of indexes over the years and where do we think we sit today with regard to all that history?

Ryan Smith: yeah that's actually a really fascinating question that i try and um i try and ask myself that often because it's very tempting like you can get into these um these tunnels of optimization where you think like you know like for example with um with the tool that i'm currently working on at index supply shovel which is you know um you could use shovel to build something like mint.fun right. so so i i spend a lot of time working on shovel and it has an opinionated way of like how to index ethereum data and it's in the tunnel that i could get trapped in. is you know trying to optimize that particular flow where instead you know or or sometimes it's good to ask wait a minute like what if ethereum was just built differently like what and i think like shadow for example. um i i know there's. there was probably some controversy around some of the stuff that they're wanting to do but i generally think that that type of thinking is like very good because it's like always fun to like go back and just say like wait a minute why are why the hell? are there logs in the first place and like? is there some way you know to borrow the parlance of um of our community? you know it. can we enshrine or or in other words like? can we just reimagine a different like you know a different foundation that would make stuff like indexing you know easier? um and i i don't have i don't have an answer to that question. i think if i did i'd probably be working on that. maybe this is just like a bad answer to your question. but yes we should think about we should think about it and i think people are and i think the answer is not obvious. and and to comment more on like what you know why indexing at all? and and are things okay? and will things just? you know you know is is the situation you know sustainable? i think the answer to that question is yes um and this could be a little bit of a cope but it's good to you know when you're an ethereum node you have to think about you know like censorship resistance you have to think about you know very quickly you know participating in a decentralized p2p network. you have to think about like you know keeping a map of the nodes on the network. you have to think about broadcasting transactions. you have to think about a mempool. you have to think about maintaining the the state. you know like the the merkle patricia tree things like this. those types of problems are different than i want to. you know query the latest. you know mints that happened like those are orthogonal problems and so i think it's seems okay to me to say okay we're gonna like build an ethereum node and it's going to be optimized for you know like keeping the network online for processing transactions for maintaining the state and then uh and then also you can add another piece of software and this will help you like morph those core data structures into something that is more easily. you know that you can query on more easily. i think that's okay. like that doesn't like. that may not be ideal and there might be a better solution out there but this doesn't strike me as a terrible solution and so i'm okay. so i'm okay with it. uh i don't think it. i haven't seen a better solution

Nicholas: and i suppose by decoupling the database architecture from the blockchain itself not only do you allow the developers the blockchain to focus on the kind of consistency and security questions that are most important for them uh but also you get a variety of different types of solutions. i was wondering if maybe before we jump too far into index supply solution can you maybe stake out like what are the different ways of looking at the blockchain? we talked about rpc. you mentioned briefly the graph. um how do you divide these different solutions? maybe hosted apis also?

Ryan Smith: yeah i mean basically you know there's there's really just kind of one way to do it and that is like you run an ethereum node. you know the ethereum node whether it's geth or it's rest or nethermind or aragon. you know each one of these nodes uh has an internal database. um ref uses mdbx geth uses um a cool uh database called pebble from cockroach db. and uh shout out to uh jackson who was um an engineer at chain uh an absolute uh legendary engineer and he is now actually the maintainer of pebble at cockroach and so i think it's quite hilarious that you know he tried to leave the blockchain space but he ended up working on like the project which is the primary data. so that's i always get a kick out of that. but exactly um. but so anyways these nodes have their own internal databases that are optimized for the operations of the node and then you have you know like your your node api providers um like you know alchemy or quick node or whatever. and um you know they're basically just running those nodes and then they're building you know um really efficient. you know http interfaces uh so that people can use you know like the json rpc clients to like access data from the nodes. and then these companies may also do indexing. right like you know they might have like a specialized nft api or they might uh have you know some sort of um you know clever way of uh accessing like you know a state of a contract or something right um but at the end of the day it's like well they're they're going to run a node and then they're going to like you know offer an api to that node and then they're going to do some indexing on the side and then offer an api to their index data. you know at the end of the day it's like well there's really only one thing you can do and that is like you run a node or you have someone run a node for you and then that provides you you know some level of access to the core ethereum data structures.

Nicholas: so you would either run a node like locally uh on prem maybe on your local machine or you might run it on aws or you might use something like quick node which i guess is just like a specialized uh kind of interface on top of something like aws.

Ryan Smith: exactly yeah it's like. you know you can outsource like if you don't want to run a node which i mean in this in the l2 world you know if you wanted to be on all the l2s like it actually it ends up being like a huge pain in the ass to like run all the nodes um and so that's where it's like. hey let me just like outsource this and i'll you know i'll talk to a company and get you know i'll access the node through their api.

Nicholas: but there is also a little distinction. not that it's that relevant for the indexing story but there is a bit of a distinction between something like quick node where my impression is that you're running your node on their i mean even not you know some aws or something that they license. they uh you know subscribe to your node and then you're running your node on their on your behalf but uh versus something like infura or alchemy where it's maybe a half step or a full step more refined in terms of api access. that is more like alchemy especially offers all kinds of like nft indexing apis and things that at least my impression is the core competency of something like quick node is not that i want to be careful.

Ryan Smith: i i don't know all the fine details of all the different node providers. like i would have to like sit down just like map it all out on some on a text file or something. but but basically like you know ethereum the ethereum node has a standard json rpc http api and if you're like the bare minimum that you need to be like considered like a node hosted node provider is like you need to at least have that standard api. and then you know these node providers are going to offer additional apis like on top of the standard where they might do some special. you know nft indexing or whatever and they can kind of differentiate themselves that way. so it's like you know. they're running the node they're given the standard api and then they may also be offering some sort of. you know they may be indexing their data in a way that is unique to that company and then they can offer apis over that index data got it right.

Nicholas: so and then there's also the graph which is a different way of looking at the blockchain. right. i mean i guess there are nodes underneath but there's more going on in that system.

Ryan Smith: yeah and so you know i think that the graph is kind of certainly one of the first like you know open source or solutions that where you could say hey i want to like process you know all of the ethereum database like i want to get all those access to all those logs i want to decode them i want to look for particular values that are interesting to my application and then i want to save them to a postgres database that i can query. uh and so the the graph definitely solved that problem and i think that's kind of. that's been the default answer for for a while. and and so you can you can take the graph you can point it at your own ethereum node or you can point it at a hosted ethereum node you know ran by quicknode or something like that and you can also use that to or Alchemy or whoever. And then you can then have that Ethereum data that comes from those hosted node providers and you can write it into your own Postgres database that you can then query. Or, you know, the way that your app may query it could also just be through, you know, GraphQL, through the GraphQL interface. So you might not even have to touch a Postgres database. If you're an application developer and you just like connect the graph to, you know, QuickNode and then, you know, you have your GraphQL client in your app and then you just send those requests to your Graph node and then you get the Ethereum data that is, you know, specially tuned for your application.

Nicholas: What kind of applications would require that you have your own, like, directly access to Postgres database versus just doing GraphQL queries to the graph?

Ryan Smith: Well, I think any time that you care about... I want to be careful with how I word this, but... Okay, so we're definitely... This is more like Ryan's opinion. Unless, like, let me, you know, lay out a matter of fact. But my opinion is that, you know, when you're building software, you can just, like, grab a whole bunch of off-the-shelf technologies and you can, like, string them together and, like, you can probably solve your problem. In that world, you run the risk of introducing a tremendous amount of complexity and that complexity will, I believe, in, you know, again, Ryan's opinion, I believe, you know, by adding a lot of complexity, you end up just, like, hamstringing your project over time because you eventually will run into, like, performance issues or, like, reliability issues or, like, the system is doing way too many things and it becomes super complicated and then the software just ends up being really shitty. And so the school of engineering that I was raised in and, you know, basically there were some more experienced and amazing engineers at Heroku that I basically just became, you know, they were my... I was their protege and they were my mentors. I learned a lot about how to build just, like, really simple software. and actually there's this really great talk by a legendary programmer named Rich Hickey who was the inventor of Clojure and he was a C++ guy for a while. and he also invented the database Datomic. and he has this talk called Simple Made Easy. and it's a fascinating talk where he just explores, like, the notion of, like, of simple and easy and how simple actually isn't easy and how sometimes people can conflate them and he even gives, like, a mathematical definition of, like, complexity. and just a really fascinating talk and I think it kind of helps inform my perspective which is I think that you want to have the simplest system that you possibly can if it matters. Like, if it's, like, a weekend hack project then who cares? Just glue a bunch of stuff together. It'll be easy. It might not be simple but it'll be easy and it might get you where you need to go. If you were building a business and you want something, like, simple and reliable and fast, and, like, well understood then that requires, like, being more detail-orientated and being more careful with how you lay out data in your database. and so this is where, you know, coming back to your original question, this is where, like, if you're in that latter camp where it's, like, you want precision, you want efficiency, you want reliability, then you're going to want to put your data, like, honestly, you're going to want to put your data in a Postgres database. Like, there's solutions like PlanetSkill out there which aren't Postgres but are offered, like, good tools. I still, like, I'm not convinced. I still think that Postgres is the best solution for most problems when it comes to, like, building web apps especially but you're going to want to, you know, design a schema that you can query efficiently. that isn't going to cost you an arm and a leg and you're just going to have to be judicious with your SQL queries and stuff like this. but I think that in that world you end up with, like, a really, you know, solid foundation to build a company on top of and so this is why I think this is, now we're getting really close to the motivation behind, like, the details of index supply which is, like, I want to build tools for this type of thinking where it's, like, okay, I want to really efficiently get data out of Ethereum and I want to get it into Postgres in the simplest, most reliable, and fastest way that I can and then that will give me a good foundation then to build queries on top of that Postgres database so that I can build, you know, a fast, responsive, reliable application like the .com portal or, like, Dymo. Yeah, this is Shuffle, exactly.

Nicholas: So, when I'm using Shuffle, maybe, could you describe a little bit what the workflow is if I want to set up or, actually, maybe even before we jump into that I guess the first question I have, really, is as an app dev or from the perspective of an app dev because I guess that is kind of the customer, right? Someone like Context or... Totally. So, from the perspective of an app dev and I also think it's easier for people just to relate to things in terms of app development. is it something that... Should people be thinking about, using something like Shuffle or, like, building their own databases for their applications before they achieve PMF or after they achieve PMF? Do you think it makes sense to build this into your prototyping? Like, I guess a company like Context slash Mint.fun? they know that indexing is going to be a core competency for them no matter what they build. It's going to be built on top of some indexing. So, they maybe need some kind of indexing solution in the course of building whatever variety of products they might pivot through as they explore the idea maze. But, do you think for an app dev it's something that people should be thinking of like, very early on? Or, does it depend on the product? How do you think about this in terms of order of operations?

Ryan Smith: Yeah, so I... Yes, I think so. I think I would recommend that people do think in these terms when they're just starting out. Because I believe that there is a very simple framework of thought that you can adopt. And once you adopt that framework then I think it makes it very easy to iterate and to build reliable, simple applications on top of Ethereum. And Shovel is different than the graph. It's different than other competing solutions out there in that the only thing that it does is it takes events that are emitted through logs of a smart contract which are structured. It takes those structured logs and it simply copies them into a Postgres table. So, for example, an ERC-20 transaction has a log that happens when the transfer happens and it has from, to, and amount. Right? And so you can imagine a Postgres table called transfers and it has a column from, to, and amount. And from that schema, if you just think in those terms like, okay, I take the event and map it onto a Postgres table and then now you can think like, okay, if I need to build a balance for a user then I can go sum up. I can sum up the rows where the user is receiving money where their address is in the to field and then I can sum up where they're sending money or their address is in the from field and then I can subtract those two numbers and then now I've got their balance. Right. So I think that's very helpful. to just think in terms of like, okay, there's this famous, are you familiar, have you heard the computer science author Fred Brooks? Have you heard of this guy?

Nicholas: No, I'm not sure if I have, no.

Ryan Smith: Have you heard of the book The Mythical Man Month?

Nicholas: Yes, yes, I have.

Ryan Smith: Okay, so that is Fred Brooks and he has this amazing quote from like, you know, IBM 1950s era. but he says like, show me your flow tables and I will continue to be mystified. or I'm sorry, show me your flow charts and I'll continue to be mystified. Show me your tables and I won't need your flow charts. So basically, if you have like, you know, the first thing that I do when I pop onto any project is I'm like, show me the data structures. Like, what is the, what's the schema look like? And then if I can see the schema, then I can guess what you're going to do with the data, right? Like, I can guess what your functions are just by looking at the schema. And so I think that this is a very helpful place to start. if you're building an application on top of Ethereum is just like, well, what schema do you need to have? And then, you know, what data is available to you? And this kind of framework of thought is okay, it's like, well, you know, I need to have this schema so I can do these queries, you know, to build my interface. And then the data is coming from Ethereum and it's going to come from these log events. And so I have these log events and if they could just, if they could just appear in a Postgres table, then I can query that table and I can build my interface, right? So, and this is what Shovel does. Shovel basically, you know, allows you to say, these are the events that I'm interested in and I want them to map onto, you know, it's like I have X events and I want to map those X events onto Y Postgres tables. And so please, and then you just tell, you know, Shovel, say, please do that for me. And so then Shovel, you know, connects to the Ethereum node, it decodes all the data, you know, and then it just like simply copies it into your Postgres table. So then from the application developer's perspective, it's like you now have like a set of, you know, tables, you know, be it like mints or transfers or seaport sales or whatever, right? Like, you know, that you just have those tables available to you and they are being populated as the blockchain, you know, moves along. And so then you can then query those tables to get exactly the type of Ethereum information that you want.

Nicholas: So each event type will correspond to its own table.

Ryan Smith: Yes, it can. Now, I want to say that, like, there are some nuances, like you might say like, oh, I want to have two or three different events that both go into a single table, you know, because maybe that design makes more sense for your application. It might be more efficient that way. So that's possible. But yeah, at a very high level, it's like, yes, I want like, you know, each event, I just want to see that data in a Postgres table.

Nicholas: And so I'm tempted to make a comparison to the graph here. Is there a, is the notion similar to a subgraph where you're writing a little process that will be applied across each block as it indexes the entire chain? Or are you then, are you, how do you think about it? Does it relate to that? It is similar.

Ryan Smith: And again, this is where my opinion kind of comes in here. But I actually don't think that, like, I was never happy with the abstraction of the graph.

Nicholas: And I'm not going to say that. From your expert perspective, how the graph abstracts this and then describe yours?

Ryan Smith: So when you get, when you're writing a subgraph, basically you write some JavaScript code and that JavaScript gets, code gets called every time that it sees a particular event happening on the blockchain. So you might say like, hey, I'm interested in, you know, ERC20 transfers. So whenever an ERC20 transfer happens on Ethereum, please call my JavaScript code and my JavaScript code will do some stuff. And then my JavaScript code will then also write, you know, using a graph abstraction, it will write data to a Postgres database. Okay. So the problem that I have with that is if I'm looking at an Ethereum application and let's say that like, you know, there's like React Native. So I have to like look at some, you know, front end code. There is like an API that the client talks to. So I have to look at, you know, maybe some TypeScript node server code. And then I have to look at, you know, the database. And then also I have to go and like look at all of the subgraph code. And so I have to like go inspect, you know, my subgraph code to see what the application is doing. And to me, like now what you've done is you've spread your business logic across a component that where it ought not to be. And so where Shovel is different, like my difference of opinion is that the indexer part, its only job should be copy these events to these tables. And there's no custom logic. That's not where you get to put your custom logic, where you get to like make contract calls. Like you do a whole bunch of other stuff, right? Like don't do that. Instead, just copy the data to the Postgres database. And Shovel has some like a declarative language where you can specify filters, you know, like let's say that you're only interested in a particular contract, or you're only interested in events where they come from a particular address, you know, stuff like this. So you can specify these in Shovel and you don't have to write any custom code. And then Shovel just like puts the data in the database for you. Then in your application, if you want to do things that you might have done previously in a subgraph, you can do that, but just do it entirely within your application. So now your business logic isn't spread out across several different components. Your business logic is just in one place. And so the indexer then becomes much simpler because we've removed some of that custom logic and we just like kept it inside the application because that's, you know, that's the unique part of your application. It should live in your API. Don't put it in your, don't put it in a different system.

Nicholas: - So Shovel is, when you say that you can apply its filters and things, those are things that you would do via like a command line or there's some kind of configuration files?

Ryan Smith: - There's a configuration file. Yeah. So, yeah. So when you set out to Shovel, you have a JSON file and you basically, you know, bring over your events just through the JSON API format. And then you just say, you know, for a particular event, you say, I want this field. I want that field. And I want them to map onto this table and I want them to have these column names. And then you can also say, and I only care about this field. I only care about this address or this ad, you know, like you can apply these filters, but it's a declarative JSON based language. And that's important because once you have that, then you can just automate all of this. And like, for example, Shovel has like this web interface where you can just drag an API JSON file onto the web interface. And then you can just use your mouse and say, I want this event and I want these fields in that event. And I want them to go to this table and then you just click go. And then it just happens, right? So there's no, there's many less foot guns. And then that also gives, shovel the opportunity to just be really reliable and really focused on performance because I don't have to worry about people. It's like the foot gun is, well, in your sub graph, you could have like code that does like, you know, buggy things or it could- in JavaScript. And so now your indexer is like caught up in all of that. Right? - And so to me, that's just like, that wasn't the optimal design from my perspective, 'cause I just want the indexer to be as simple and as brand new as possible. Just like, please just copy the data into my Postgres database and then I'll take it from there.

Nicholas: So we're shoveling the data from a node that we're running, maybe locally or somewhere else into Postgres database. And then I suppose you do some other transformations, creating new tables for things that you might need for your application subsequently, but it's not involved in the indexing process. It's sort of subsequent.

Ryan Smith: - Right, exactly. So for example, let's say that you wanted to have like some sort of leaderboard, you know, and you wanted to compute, you know, who has minted, you know, the most NFTs in the last hour or whatever. And you wanted to have this thing kind of be up to date. Well, what you would do is inside your application, you would just like go, you know, every time that a block happens, you can go query your Postgres database. You can do these computations that like produce your leaderboard, and then you can save them to another table. And then you can just do this. You know, the great thing about this blockchain architecture is that, you know, we have like this constant ticker, which is a new block, right? So on the L2s, it's like, you know, every two seconds, you know, on L1, it's every 12 seconds. But basically we always have like this ticker and inside of our applications, we can just say, okay, next time that there's a new block, which I can just see by looking, it's a very cheap call to like query your Postgres database, be like, oh, there's a new block. So then on that new block, I can say, okay, go take the leaderboard per the last block, add in all of the new events that we've seen recently, and then save that, you know, to a different table. And so when someone, when a user opens the website and they want to look at the leaderboard, now I'm just doing a single query on a leaderboard table, which is very fast and it's pre-computed. So it should be very scalable and very efficient. And so my thing is that like that logic should be inside your application. That has nothing to do with the index, right? Like that's like the leaderboard is like your special sauce. So don't mix it in with someone else's infrastructure.

Nicholas: - Makes sense, makes sense. So I guess this lets Shovel be very simple. And I mean, are people using Shovel today? Is it in production?

Ryan Smith: - Yeah, I've got a handful, I've got a handful of people. I've spent the last month or so really just working with teams. And one of the things that I do with Index Supply is because I'm, and we haven't even gotten into this, and this is like a whole other topic, I guess we could talk about, but Index Supply isn't venture funded. And, you know, per what I was mentioning earlier with just having a more longer term vision, I wanted to build a developer tools company that could be very sustainable. And so I'm very careful with how my incentives as the, the founder of the company align with my users incentives. And so because of that, I've chosen not to pursue venture. And instead I sell support contracts for Shovel. And then I also, right now I'm doing integration. So like if you're a project, you know, that just needs, you know, like you want to fast forward and just like build like a really good, you know, foundation to build your app on. I can just like help you basically set up Shovel. And I can like, you know, I've worked with like, for example, Dymo. Like I've been very close with that team. They're an amazing team. And I was able to, you know, go in there and kind of help them sort out, you know, some of the indexing things that they wanted to achieve with Dymo and Shovel was a part of that. So, you know, I spent some time getting Dymo to use tables that are written to by Shovel. And so that's one example of someone who's running Shovel in the wild right now. And there's a couple others too.

Nicholas: - So, okay. I have a bunch of different questions about this and we will get back to the VC, non-VC choice in a little bit. But before that, so something like Dymo. So Dymo is a multi-chain, I guess, L2 centric, stable coin wallet with pass keys and smart accounts, if I remember correctly. What are the challenges that they're facing? I guess the multi-chain aspect is a big one.

Ryan Smith: - Well, you know, and in fact, one of the beautiful things about Dymo is like it's simplicity, but also both DC and Nolan are just, you know, we're actually just some of the founders of Dymo. They're just incredibly brilliant engineers and builders in Ethereum. And they have an incredible depth of understanding in terms of like UI, UX, but then also they've done just amazing things at the protocol level. So they've built like a really secure payment system on top of ERC-4337. And their indexing needs are very simple, actually, you know? - Yeah. the, the, the, the most interesting parts of their application, like are not indexing at all. And in fact, they just wanted, you know, again, it's like, there's a set of like events that they cared about and they weren't even like critical to their application. It was more like, um, because you know, their application, like the critical parts of their application are actually built into, you know, the, the contracts that are running on the chains. Um, but when it comes to like building like the UI and like having, you know, a really good interface around the wallet around the payment apps, um, you know, they just wanted to have a set of Postgres tables, um, that basically mirrored, you know, the data structures that were being manipulated on Ethereum. And then they wanted very fast access to those data structures and they wanted a very simple, reliable system to do all that. Um, and so that was kind of the appeal for shovel was like, Hey, let's just like, you know, let's take these set of events that you care about and let's just map them onto Postgres tables. And then in the Dymo API, uh, you can just, you know, query those Postgres tables, uh, whenever, you know, you have a user request that you're trying to, you know, answer about like transaction history or like if you rotated your keys or, you know, stuff like that. Hmm.

Nicholas: Uh, very interesting. Uh, we kind of, I guess it wasn't the multi-chain piece. Wasn't a big problem, but I am curious, what, what is the story with multi-chain and shovel? How does it, uh, how, how, how has multi-chain complicated things or not that much?

Ryan Smith: So multi-chain is complicated things for people who want to run. Their own notes, because it used to be that like, and also before the merge, you had like to, to, to run an Ethereum node, you just ran Gath. After the merge, you have to run Gath, but then you also have like, you also have to run a consensus client. So, you know, you have to run two things now, uh, to have just one Ethereum node, um, on the network. And then with L2s, it's like, well, you have to copy that for each L2 that you want to participate in. So if you want to do, you know, main net Gourley base and base Gourley, it's like, you, you have to copy that for each L2 that you want to. now have to run eight pieces of software, which I don't think is, you know, that's, that's a tall order. Um, so I tend to recommend my, you know, people who use shovel, I'm like, unless you have a very specific use case and I'm a, I don't care. I'm the biggest proponent of like. run your own node. But at a certain point, it's just like, geez, unless you really need to, like the costs are high right now. And maybe that will change in the future. And I think that that could be a, a cool avenue to explore even for index supply is what can we do to simplify that?

Nicholas: So what, what is the easy answer for now?

Ryan Smith: So the easy answer is like, you just need to use like quick node, for example, and then you can get like, you know, you can get main net, you can get base, you can get base Gourley and then shovel a single shovel. You can, you can say, Hey, I care about these events and I want to index them across these three chains. And so you just go to, you know, quick node, you boot up three end points for each network that you care about. And then you just take those URLs and you stick them in your shovel configuration file. And so now shovel a single shovel can, you know, go to all three of these, you know, Chains, or how many ever you have, you know, you can go to all your chains that you care about and it can index that data and then they can put it into your Postgres table. And then, you know, you can have a transfers table that has a chain ID column on it. And so then you can, you know, say, you know, you can query all the transfers that happened on base or query all the transfers that happened on base and base Gourley. Um, so, so shovel handles.

Nicholas: So an application like application like Dymo would be making essentially one RPC request per block per chain total.

Ryan Smith: Yeah. Yeah, basically. Yeah.

Nicholas: Hmm. Interesting. Aside from, I guess the rights they need to, well, it depends if they have an integrated wallet or something, but depends on how, how, how the transactions are propagated to the chain. Maybe they need to do more, uh, RPC requests in order to write, but, uh, A hundred percent.

Ryan Smith: Yes, that is true. Like the actual, you know, process of making payments is actually just a separate thing from indexing.

Nicholas: Right. Right. Uh, but I guess it'll depend if they're like, I guess Dymo is a passkey. I mean, it's kind of like an embedded, uh, wallet. So you're going to, they're going to need to be the ones propagating to an RPC versus something like, uh, winded on Ethereum, like a MetaMask Rabi style wallet would be, have its own RPC connection. that would, uh, cover the cost rather than the DAP itself. So there's kind of an interesting difference there for app devs.

Ryan Smith: As it relates to the particular details of how Dymo works. I don't want to, um, I don't want to speak too much on their behalf because I know it's very particular and I think that they have done a pretty good job documenting it. So for people who want to learn exactly what's going on with, With how you transact on Dymo, I'd just say, go check out their docs or whatever, but I will say that it's, it's very good.

Nicholas: That's what I'll say for sure. Also there's a, they were on the show at the beginning of November. So if you want to hear DC, you can check out a web three galaxy brain.com and there's a transcript too. If you want to take a look, search for what you're interested in. So, so, so how, so basically a dev thinking about the costs of running something like shovel, it really reduces the RPC reads like a lot. You basically only need to do one per block per chain that you're interested in. And then you're dealing with everything inside your Postgres databases, transforming it for your API, and then just making calls to your own API from the client. So you say potentially say precisely, yes. And for people who are going and not going the shovel route or not self-hosting their own indexing solution I guess the costs can be quite high, both for making RPC calls, but also for making more sophisticated API calls. I guess there's even more probably markup on those. Kinds of calls than just the straight RPC calls, which are kind of commodity at this point. Right. But then there's other things like when I think about mint.fun, I think about, they have that feature where if you're looking at a collection and someone has a, there's like a mint in flight, like in the mem pool, they surface that. I think JEM maybe was also showing this back in the day. But that's the kind of thing that I guess you would have to have like a ongoing connection with an RPC. You couldn't do that from a database because you're not updating it per block. That's happening sort of unrelated. Exactly. To the block time.

Ryan Smith: Well, so that if you want to build a feature like that, each full node, and I'm not sure where the hosted node providers are in this particular offering, but if you have a full node, which, well, I don't know how much I can say about mint.fun's architecture, but I will say this. If you would like to build a feature where you want to show transactions that have been confirmed yet that are in the mint pool, like the in-flight transactions, if you want to build like a little spinner or something, or you want to show how many similar are in the mint pool, you can do that very easily just by talking to an Ethereum node and you can ask it for all the transactions in the mint pool. And then you can copy that. The way I would do it is I would just talk to a node, get the mint pool, copy that to a Postgres table, and then do that on some sort of schedule, a short schedule, you know, like every second or so. And then, and then as a block lands, you know, some of those transactions will be removed from the mint pool and they'll be confirmed. Right. And so then each time that a block lands, I'll just go and clean up all, I'll go and remove all the transactions that just got confirmed. So that's like one example of how you might build that kind of feature, just using, you know, using the API that a node gives you. And like I said, I think that some of the node providers offer this, not, not totally for sure, but if you just have like the, you know, any standard node, you can build that feature.

Nicholas: Right. So you're not, I guess one takeaway is that you're not married to the blocks that have been incorporated into the chain, but you have the full latitude of everything provided by the node software. You're not, you're not restricted to just the chain. Very cool. I noticed you're also able to do like concurrent indexing. Maybe you could explain a little bit about how that works and I guess the speed advantage of that.

Ryan Smith: Yeah. So when it comes to like backfilling, you know, like you, you might say like, Hey, I'm interested in a particular event and I'm also interested in this event as it happened throughout history. And so, you know, to block, you know, 1 million or something. Right. And I want to, you know, I want to have all of that data in my Postgres table to do that. You, you know, like, and it, you kind of don't want to wait forever. Uh, and this was like, I think one of the gripes that a lot of people have with the graph is that it's not optimized for this particular use case. Um, and so it can take a while. Uh, and so that can be kind of frustrating experience, even though it's like under the hood, it's like, well, you're processing, you know, a lot of data. So it's, you know, it's definitely like a computationally expensive problem to solve, but yeah, one of the things that shovel was designed for, um, was just like, it'll use as many cores as your machine, like the, the, the machine that you deploy shovel to, like, if you give it a bunch of CPU cores and you give it some good bandwidth, um, it can process a lot of stuff concurrently and in parallel. Um, so this makes backfilling, uh, really nice. So, um, you know, you, you can do things in a matter of minutes that, you know, would otherwise, you know, maybe And, and I think the, the, the, the innovation there was, you know, again, coming back to like the, the core philosophy or the difference of opinion, it's just like, well, the, the indexing stuff doesn't have to do a whole bunch. Like it's, it's job is just really copy it into Postgres. And then you can kind of go off to the, you know, the races. And so, so I can, I can do, you know, I can process stuff out of order. I can, you know, I can do these types of, uh, tricks that like allow me to just massively, you know, reduce the time it takes to copy the data into your Postgres database.

Nicholas: Very cool. One other thing that came up, uh, as we talked last time and also in the docs is this difference between, uh, OLAP and OLTP. Uh, and I guess one comparison or contrast might be between Shovel, uh, being, is it OLTP and Dune, for example, being OLAP. Uh, maybe for people who are not as versed in this, could you explain a little bit what OLAP and OLTP are and what the advantages of one are over the other? Totally.

Ryan Smith: Yeah. I, the, the key difference is like the A and the T, uh, T for transaction processing, A for, for analytics processing. For example, um, one project that I personally really enjoy is Clickhouse. Um, and it's a columnar database. And what's cool about like columnar database technology is that you can do aggregate queries across massive amounts of data, uh, very quickly, um, because they optimize the way that the data is laid out on disk so that when you do aggregates on particular columns, you know, you can just very, like your machine can serve that data very quickly. So you, like, if you want to like count, you know, all of the, you know, I don't know, NFTs from like particular wall or something and like, you know, and, and you had this across like every chain and you want to do it across history, like a columnar database would be really good at like answering questions like that.

Nicholas: Yeah. On their site, they say, uh, query billions of rows in milliseconds.

Ryan Smith: There you go. Exactly. Who, who wouldn't want to do that? That sounds awesome. Um, uh, yes. And so, and so yeah, tools like that are great when you want to do, yeah, general analytics queries. But if you wanted to, like, like, for example, like. if you wanted to do like the leaderboard stuff, uh, or, or you had particular flows inside your application where you were, you wanted to, let's say you wanted to read a couple of tables and then you wanted to write the result of the data that you just read to like a, you know, like a leaderboard table, for example. Well, in that case there, boy, it would be nice to have a database transaction, meaning that like, I want, you know, and, you know, there's like several components to, you know, transactional database basis known as acid, uh, a, the first one, a for atomicity. It's like, I either want everything to work or I want nothing to work. So it's like, I want to read a couple rows from this events table and I want to compute my leaderboard. And then I want to save that leaderboard, um, to, you know, a leaderboard table. And then I also want to go like, you know, write out, um, you know, some, individual user statistics for some of my top users on the side or whatever. So you want to do all of that. And it would be nice to use a database transaction where you don't have to worry about getting into the problem of like, well, what happens if you, you know, read from your events table, you can write the leaderboard, but before you wrote your individual user statistics, your, you know, your API crashed or whatever. And so now like your database is in this inconsistent state, right? Like, and so to work around these types of issues, have this atomistic atomicity guarantee, which says like, well, either it all worked or it did, or none of it worked. And if none of it worked, then I can just retry it again. And I know my database will be in a consistent state. So when you want to do stuff like that, you need to, you know, have an OLTP database and namely you need to be running, you know, Postgres or MySQL or Planetscale or whatever. Um, you know, ClickHouse isn't going to help you solve those problems. for people who are building, you know, apps where, you know, you need to be able to have that tight level. Like you want to, if it's like a wallet, like you really want to make sure that you're the data that you're showing your users is like consistent and that you're not like corrupting, you know, metadata along the way, like that data needs to be in Postgres. Um, but if you're just doing like these massive, like analytics queries where it's like, Hey, the query was a few seconds out of day, or, you know, it's like, okay, not that big of a deal. You know, that's where those OLAP solutions tend to work out better. Um, so that's kind of like a rough, rough, rough, rough, rough, rough, rough, rough, rough, rough, rough, rough, rough, rough, comparing contrast.

Nicholas: So OLAP, uh, is more suitable for something like Dune, uh, or for ClickHouse because these kinds of large analytics queries don't need quite as much precision as, uh, something that might be like business logic for queuing up a transaction or giving you actionable data. that needs to be very, very precise. Is that a totally exactly.

Ryan Smith: Yep. That's a great summary.

Nicholas: Uh, all right. Fascinating. So I, we sort of did a round, uh, you know, tour of shovel, but that's not all that. you're working on an index of play. You're also working on the block API. So what is the block API? How does it relate to shovel? Okay.

Ryan Smith: So, um, shovel will connect to any Ethereum node or Ethereum EVM compatible node using the Jason RPC interface. This is the standard interface that we've been talking about that all the node providers offer that geth and wrath and nevermind that they all offer the standard, you know, enshrined, uh, Jason RPC API. Uh, Jason is, uh, and then we've also talked, let me pause it. We've also talked about, you want to backfill, right? And like you, you, you want to say like, Hey, I want to create a Postgres table that has all of the ERC 20 transfers. And I want to, um, you know, filter it by this particular address and, you know, please do it across, you know, these, you know, set of chains. And you want that to happen fast. Cause it's the, you know, 21st century. It's like, I want to, I want that to happen right away. Well, Jason, like if I just put on my first principles hat and said, okay, I'm going to like solve this problem of like, I have data. You know, internal Ethereum node database, and I want to get it out of that format. And I want to put it into a slightly different format and into a different database. I want to put it into my Postgres database and I want to have it indexed by a particular column. Okay. If I were just like first principles thinking through how to solve this problem, I can guarantee you that I would not use Jason. Jason is a terrible format for this type of operation. Instead, what I would want to do is I would want to minimize every, like every transformation. That I do is it has a cost associated with it. And so I'd want to minimize the amount of transformation that I do. Ethereum data is, uh, it, it has a canonical bite representation on desk. And it's, um, there's been some debates over where the future of this is heading, but today, and for the foreseeable future, that canonical format is called RLP recursive length prefix, which I believe is something that metallic invented, uh, at the same time, uh, like it first appeared in the Ethereum yellow. So it's, and, and I know that people kind of don't like it for various reasons, but the fact is there is a compact and compressible, uh, bite representation for the raw theory of data. And so if I were going to, if I were tasked with the problem, my boss came in and said, Hey, we need you to get the data out of the Ethereum node and into the Postgres database as fast as you possibly can. I would, my first thing I would do is I say, okay, well, let's look at what format the data is already in. And then maybe we can, you know, just massage it ever slightly, uh, to get into this, this different format so that we can stick it into Postgres. And so that's what the block API is. The block API is me saying, okay, I know that we have the JSON RPC API and shovel will always work with that because I believe that it should be like, I believe that you should take the open source shovel product and you should be able to connect it to your gap node, or you should be able to connect it to, you know, quick node and then, you know, time data into your Postgres database. That should always work. But also if you would like to use a hyper optimized API so that you can Even faster. Let me pause really quickly and say in order of speed, uh, the fastest, the fastest thing that you could possibly use to index data is you could run shovel on the same computer as a full Ethereum node. So if you like booted up an EC2 instance, you put ref on it and then you put shovel next to it and then shovel just read from the, the hard drive that rep is using. That's the fastest. you're not going to get faster than that. That's great. The next best thing would be in my opinion, the block API, which is an API that is designed, uh, specifically for getting, so it's like, you don't run your node. Someone else runs your node, but it's in a format that is optimized for indexing, right? That's the block API. And then the slowest, uh, that we know about today is would just be like a JSON RPC API. And you might say, well, why did they have JSON RPC API? Well, JSON RPC is nice. Like in a browser, it's nice. It's like, it's easy to use from JavaScript. You know, like I'm not saying that the JSON RPC API is just carte blanche bad. It's just not what you use. If the pro if the only problem that you were trying to solve is like getting data out of the Ethereum node into a postgres database. And so the block API is like this niche thing that I'm building because I just wanted an API that was optimized for indexing only. So even if you use the block API for indexing, you know, and you were building like a payment system, like you probably still would have like, you know, an RPC JSON RPC API that you would also use so you could submit transactions. So you could, you know, do just various contract reads. But when it came to indexing, like you could have a special purpose API just for that.

Nicholas: And why would I use block API over shovel?

Ryan Smith: You would use block API with shovel. So shovel can connect to the standard JSON RPC APIs, but if you want something faster and you don't want to run your own node, you can connect it to the block API.

Nicholas: Ah, okay. And the block API underneath, where does it pull its node connection from?

Ryan Smith: So the block API, it like, so index supply me, uh, I'm running, I'm running Ethereum nodes. And I have the, and the block API, what it does is I deploy, this is more like a peek under the cover of how does the block API work. So basically, I write software that I deploy and I, and it runs on the same machine as a full Ethereum node. And when shovel, like if shovel is connected to the block API and shovel says, hey, please give me, you know, these blocks, cause I'm, I'm trying to index these events. So the block API, I will say, okay, I'm going to go quickly. Read that data in the binary format from the Ethereum nodes, low-level database. And then I'm going to send that to you without doing me, you know, I'm not going to put it in JSON format. I'm just going to send you the raw compressed binary data. And so they, and that's like the fastest way that you can give someone the data. Right. And so then shovel will then just continue operating as it, as it would, um, you know, indexing that data. So, yeah, so it's like, it's just like an opt, it's like, and that's like, you know, one of the goals, you know, to generate revenue for the company to fund the open source development. shovel is like, Hey, we have this API and it may be, it might not be, you know, for everyone, it might not be like this thing that's going to corner the entire market, but like, if you care about, you know, getting low latency, high throughput access to like indexing, it's a good API to use.

Nicholas: And I guess solves the problem of running all these different node softwares that are required, especially in a multi-chain world. Yes, exactly. Very cool. So you alluded to it a little bit before, but index apply also has this non-traditional funding structure where you've issued a VC in order to take this kind of slow or, uh, more thoughtful long-term approach. Um, I guess what inspired you to do that in the first place?

Ryan Smith: So I'm not like a, like, I don't think VC bad, like. that's not, that's not my position. I think VC is like very interesting and has like done a tremendous amount of good in the world. Um, but the, the VC route, you know, and, and through my experience with, you know, Heroku and then with chain, um, and then there was even a, a company called Pogo, which was a VC backed company that I did after chain that we ended up winding down. I I've been through the VC path several times. And, um, and I know, I know the, I know, I know the pros and I know the cons. And one of the things that, you know, I, I alluded earlier to the fact that like I got decentralized pill that I really like got excited about, you know, spending the rest of my, like, I love what I do and I want to do it for as long as I can. Given that it wasn't obvious to me that raising, um, VC money for the company was the right solution because I think that when I think for, I think that you can get, it's, it's hard for a developer tools company to also be a VC company. And like, um, you know, Heroku is an example of that. Um, you know, I, unfortunately like Heroku isn't like what it once was. Um, and, and, and it had to sell, you know, like there was an exit to Salesforce and stuff like that, which made a lot of money for, you know, a lot of people, but the, the actual product has kind of like, like died on the vine a little bit. I think that HashiCorp is another interesting example. Uh, you know, they recently had to change all of their licensing on their, on, on Terraform. Uh, and they changed it to this, this business licensing that like, you know, basically they went public and then now they have to like start caring about, you know, all these different revenue situations. And so then that forces them to rethink how they relate to their users and their users didn't like that. And so then there was like a fork and like it created all this drama. And I guess my point is, is that I think that when it comes to developer tooling and when it comes to like infrastructure and specifically open source infrastructure, I think that there needs to be really good incentive alignment between the people who are using, uh, the open source infrastructure and then the company that is developing the open source infrastructure. And I think that when you take VC funding, I think that that alignment of incentives break potentially breaks down. It's not guaranteed, but like you're basically, you know, dropping a little time bomb in there, right? Because if, if you raise, you know, like if, if I went out and raised $10 million, well guess what, I have expectations now, you know? And, and if I don't meet those expectations, uh, then the company is going to die. Right.

Nicholas: Um, and so And the expectation, the expectations are specifically more for an exit than for like longterm sustainable development of some reliable piece of infrastructure that others can, can use.

Ryan Smith: A hundred percent. If I raised $10 million, I'm either expected to sell the company for $200 million or go public. What if I don't want to sell this company? So I'm like, what? to do that?

Nicholas: right right what if there is a revenue generating business? but it's not? um it may be possible to convince someone like salesforce to purchase a company but then they'll have to transform it into something that generates revenue much more than maybe the original killer app. uh from that company would do so naturally over time. yes yeah i think there is a return to kind of private corporations. uh that's happening. i think people are observing it. one other example that i often think of is zapier only ever did the yc seed round and then became a profitable company and never raised again. and i think if you give away i don't know whatever five to eight percent of the company maybe ten percent at the high end for some seed round you know i think there's sort of less pressure and you can maybe buy out the investors uh rather than if you do like multiple rounds then there's greater expectations that you're gonna stick to that vc treadmill most likely 100. so so you've decided to go this other route uh and you have the experience and skills and kind of. do you know? start yourself and build the thing yourself uh to begin with. but you're also uh partnered with the zero x park in some capacity.

Ryan Smith: yes um so i i got connected uh to zero park through a friend of mine um who put me in touch with albert got me then put in touch with um brian and justin at zero x park and i basically told them everything that we've talked about today and and and they were really excited um to see kind of like a company that was that wasn't on vc path uh that was building open source infrastructure that had this kind of like long-term view on the space and and so um. so i just got along with them and really gelled and a lot of the projects that they're working on they could use um you know like shovel was appealing to them and so they uh you know they had offered a grant and uh and then offered you know just a chance to really collaborate closely across several different um projects that they had across zero x park. um and to me it just it made a lot of sense like i really enjoyed the people that i had i had met there and and honestly it's you know like it's doing the non-vc route means that like it's slow it's slow growth. right it's just me right now and i i really want to hire some some some people. i want to build a team around this but i have to do it in a very careful way because um you know it's like. well i gotta like you gotta make money you know and i i've taken like several like very i would say modest um grants um that have like kind of helped get this thing off the ground you know paid for some ec2 bills and you know afforded me some ramen or whatever. but um it's like uh yeah i i i've been i've been i've been very careful and so and so you know there's the real aspect of like man. it's it's not as it's a little more lonely than you know. if i had 10 million dollars and i just wanted to do something like that i would have done it to go like hire. you know five people to build a company. you know you have a crew. you can assemble a crew but when you're kind of like at it alone um it really makes sense to like for me to just partner with people that like i respect and that i think are going to push me into like good directions and every person that i had talked to at zero x park just i was just like blown away. i just like oh these are really smart people. they're doing really interesting things. i really like the way that they're thinking about stuff and so it was kind of like a nice little group for me to be if you don't have the the crew that a vc budget allows for.

Nicholas: but i think it's interesting to think about the kind of. i mean obviously there's a way in which vc funded things are a little bit like uh speculative gambling sort of you know put on the turbo boosters as much as possible as fast as possible machine the narrative so that people believe that it's a rocket ship and then flip it before people realize there's not that much of substance inside or um i always remember i think talking about um there's these uh european maybe german brothers the samuar brothers who would copy american startups uh. and deploy them in europe or africa or latin america or asia before the american startup would get there or what you know whatever western uh anglophone startup would arrive in that region and the a16z approach to it might have been mark adresin was. uh that the the ultimate. uh justice there. the best course of action is to let them let them have to handle that long term like they don't actually want. they just want you to acquire the company they don't want or you know or for a competitor to acquire the company. they don't really want to be left responsible bag holding their own copycat product because then they'd have to actually make a company. that makes sense um which is quite a burden. so i wonder if it's a little bit i don't know. it seems to me like there's something about this. i hear more people talking about uh sort of bootstrapping or doing things as a private company or not seeking quite so much vc. i don't know that it's going to hurt the vc part of the market. but i think it's going to hurt the vc part of the market but maybe just represents another kind of aspect of uh approaches to this kind of thing. but i think it's interesting the kind of financial discipline and uh prioritization discipline that difference when you are sort of working with a small amount of money as opposed to a huge amount of money but an incredibly short time horizon.

Ryan Smith: yeah i mean perhaps you know i'm just going to highlight some of my own weaknesses here. um but you know having having been a part of extremely well-funded um you know just like a tremendous amount of cash available to you know employees and founders and stuff. and then now i'm one year in i've been working on index flight for one year. i'm like it's very frugal. um like i've i guess what i'm saying is that like it it's really forced me into like a completely different mode of operation and i think for the better. like. i personally like myself more as like founder slash developer of this like bootstrap company than i did of like cto of you know 50 million dollar startup whatever you know like i enjoy myself more in in this mode now even though it's not as sexy and not as glamorous and like there's many dark days i it's actually like that darkness actually ends up yielding more satisfaction than like the. the vca day for me and again just personal no judgment to anyone different. this is just what i like. you know.

Nicholas: i'm curious what the difference is in terms of. i mean is it more of a charade if you're sort of having to sort of? i i hear in like. uh in the yc talks and stuff that um it's it's almost impossible for founders to raise and then not start wasting money by over hiring and thinking that just adding people will somehow solve their problems or bringing on experienced people with fancy resumes will somehow solve their problems. and i wonder is there like is the how? i wonder how in what ways the posture is different from being a cto of a vc backed organization versus a like bootstrapped you know one person or small relationships around building the thing from scratch? how does it change what you're focused on all day?

Ryan Smith: yeah so i think you know and again i like i'm i'm probably highlighting my own weaknesses more than anything because you know clearly like there are people who are extremely effective and happy at you know being founders and ctos at you know large vc strapped you know vc backed companies who like. clearly that is a real and like good thing for me personally. i think one of the things that like i really enjoy about going this bootstrapped process is it is is that it forces me to be extremely honest with myself. i think i'm forced to really think about and like i'm going through this process right now where it's like okay i'm looking at like the next year and i'm like okay you know my revenue has been extremely modest modest, you know, over the last 12 months, like I've had the grant, but like, if I just look at my revenue, you know, my revenue operations, it's like, oh boy, you know, it's like, I really would like to figure out how to like, you know, get a little bit of scalable revenue, you know, so that like, you know, I can actually see a vision where I can support, you know, like a D you know, a decent life for myself. And then I know, like, let me not say decent, like I want to, I want to have a very wealthy, you know, rich life, you know, like, that's what I want. And I'd like to have that for a few other people who joined the company as well. But I would like to do it knowing that like, we stand behind our product, and that we, we give people something that is valuable. And in exchange for that, they give us money, right? Like, that, to me is like the thing that's like what I'm shooting for. And so to be able to get there, you have to just be extremely honest, you know, with with yourself and with like the thing that you're building. And you have to say, is this actually valuable? What people do people actually want this? And which is in many ways, it's like, this is what YC preaches, right? Like Paul Graham is just like, you know, build something people want, you know, like, that, that is the YC mantra, you know, and and I just find like, I'm looking at the YC mantra now, like, from a completely different perspective, where it's like, okay, I can't afford not to, you know, it's like, the only the only way that this works is that like, if I build something people want, and like, and so it's like, I'm forced to do that. Or otherwise, I'll, you know, have to go like, you know, get a job somewhere.

Nicholas: It's interesting to me how the education that YC, I mean, it just sort of represents that there's, there's a lot of different places that are doing it. But YC kind of became a pole of putting that information all together and making it accessible to lots of people, and also kind of a culture of that. But a lot of those lessons really don't really like. VC is just turbo chargers for some existing process, unless you have a very specific plan about doing something, you know, like, some kind of loss leading activity for the longest time until you develop, I don't know, I guess, traditionally an advertising model. But it seems like even inside of YC, at least my anecdotal kind of impression from outside is that the things that are easiest to identify that people want end up being B2B services, not actually kind of huge consumer products that have made generate no revenue for the longest time or, you know, need to discover what their business model is once they have a bunch of attention. Actually, I don't think that's really what YC has kind of excelled at over time. I think it's been a handful of those. But in the end, it ends up being these more kind of predictable things where you find a customer who actually wants to buy your thing, and then you sell it to them and more versions of them across the market, which maybe doesn't even really require YC or any kind of VC, except in the case where I guess the kind of archetype of a YC founder has been someone who maybe doesn't have the capital to do it on their own. They don't have a savings account that they can bootstrap with. And so these programs provide both the funding in order to be able to do it initially, but also, I think you alluded to it also earlier, the kind of having a gang of people, a team of some sort, or even just a sort of camaraderie amongst other people who are trying to do similar things in different fields. I think that is something that maybe the bootstrapping experience lacks, at least the vanilla bootstrapping experience. without some kind of cohort you join.

Ryan Smith: Totally. Yeah. And I think that there are, you know, I do think that I am tapping into something that is already happening. Like it's a wave. That I'm not making, but I'm writing. And I see other people starting to write it too. I think like, you know, I think you're hip to this. Like, I think we're testing the meme sphere right now. We're seeing like, oh, wow, more people are thinking about bootstrapping companies. And I think like, if you just like look at the reason why it's like, well, like a lot, especially in like the crypto space, it's like a lot of us, you know, have maybe enjoyed some of the windfalls of the token appreciation. Right. You know? And so it's like, it is what it is, you know? And like, so, so take that and use that. Right. And, and maybe that's like a bootstrapping mechanism to like a whole new world. And, and so I think to the extent that people are more interested in, in bootstrapping, I think maybe there could be some cool new communities that like, kind of like pop up. Like, could there be like a cool, like an indie game developers, I think have gotten this right. And I think there's a lot of cool stuff that indie game developers have done in terms of building community. Um, but you know, I, I could see, I can totally see, you know, the, the small indie, you know, developer market or Ethereum product tinker market, like really thriving in the future.

Nicholas: Yeah, definitely. I know there's the founders channel on Farcaster that has a little bit of this energy. It also reminds me, I mentioned levels IO earlier, and he's a part of that indie hacking community, which is kind of like, you know, build something as fast and quick and dirty as possible and slap a Stripe payment thing onto it before you, or even, uh, almost like, uh, I'm also really a fan of the hyper-casual community. Yeah. Like the hyper-casual gaming market where they have, uh, like the kind of games that you see ads for on Instagram and whatnot, where they'll even do things like, um, levels levels has this thing about, uh, not writing a line of code until there's a paying customer, uh, which is sort of really extreme in the direction of testing for PMF before. I mean, maybe PMF is a little bit of an exaggeration, but testing for customers, at least before writing any code, uh, which is based on his own experience of spending two years writing something that then nobody wanted. And similarly, hyper-casual gaming market, they'll do things like, create a mock-up and then do run ads on Instagram, et cetera, of the mock-up and see click-through rate on the mock-up before they go and make the game in unity or what have you. Um, it's a little bit different in the crypto space, because I think one thing that makes the Ethereum space in particular different is that we do have this access to, because everything is tradable and because people are able to make little windfalls here and there all over the place all the time, it is possible to make these kind of small projects that end up giving you enough, uh, funds to maybe bootstrap something. Well, it'll be interesting to see how much, it'll be interesting to see if that really evolves into something very viable, because I think so far still, most of the big projects remain these kind of VC backed, uh, projects, um, rather, and there's a kind of, uh, narrative legitimacy that is inherited by having not only the VC's name on your thing, but also you see it a lot with these projects, especially in like NFT space, for example, where they'll have all kinds of partnerships with very fancy brands. And it's clear that somewhere behind the curtain, people are connecting the dots between traditional IP brands and, and fashion brands and things like that. And, uh, technology companies, whereas like a little bootstrapped startup probably isn't going to have like an Adidas partnership or what have you, uh, coming out the gate, you know, the day and date with their announcement that they raised so over many millions of dollars from a 16 Z or whomever. So yeah, it helps the meme also to be a part of that whole VC ecosystem.

Ryan Smith: It does. And, and at the same time, like the, the, the, the, the, the, the, the, the, the, the, the, I don't know, I don't know if I would call myself an anarchist per se, but like, I definitely lean that direction more than I would like, you know, totalitarianist or whatever, you know, uh, that's why I'm in the space though, is like, cause Ethereum, like the whole idea is that like, I don't need your permission to access the system, you know? And I kind of feel like that ethos, like that spirit that's in the technology is like good for humanity. Like, and I think about this a lot. More than I should honestly, and I'm I've convinced myself of it where I'm like, no, I think that like freedom is very important. And I think that being able to be free to access the chain, like to access the system is like super important. And I think that like it reduces gatekeepers. And so I think like what I'm trying to say is that like, I think the spirit of this technology is such that it enables more independent. people who maybe aren't associated with like the top tier VCs to be able to get in there and create because again, it's like, you don't need permission to like, that's the point, you know, uh, censorship resistance is the point of the technology. And so it's not perfect, but like, I feel like it's one step closer to a true free market, you know, like it reduces like a little bit of nepotism is lost in Ethereum. And like, to that, I say like, hell yes, you know, like, let's keep going in this direction. Um, and so that's kind of like another, like, you know, facet to why I did kind of want to like go down this bootstrap route. It's like, I do kind of like, I don't want to be like, like blessed, you know, it's like, I want to like build something that people use. And, and I want that to be like, you know, and maybe I don't do that. And then maybe the project dies, you know, but I like, that's, that's the stakes that I want to play in. And I, cause I think it's like, I think it's good for humans. Like, I think that's what we should do. Yeah.

Nicholas: I think it's difficult looking at the timeline. Um, it's easy to get confused between people who are building things that are trying to be real, for example, blockchain technology versus people who are doing things that are maybe using the same stack, but are fundamentally memetic products that are maybe achieving huge numbers very quickly. And then simultaneously things like VC projects that are somewhere in between, or a third world that is like private VC backed, intending to reach some kind of exit in the next five years, maybe 10 years. Um, it's easy to sort of see, the timeline is this kind of undifferentiated feed of information where everyone is competing in the same lane, but in fact, there are quite different and it's easy to get confused between something like, I don't know, a ZK project and something like a blast or something like this, more, uh, memetic or oriented opportunistic, uh, thing rather than a technology oriented project. They're not really the same thing. What's interesting is when there are collisions, like for example, polygons rise transitioning from a plasma chain, which ended up not being the correct technological choice to a POS chain, which also was a kind of stopgap bandaid solution, but hugely successful alternative to something like a Binance chain and then becoming so large that they're able to acquire dominant or promising ZK technology companies and then incorporate them into a kind of broader plan. That's interesting too, that sometimes these memetic approaches can then subsume technological approaches that fail to capture the narrative because they're able to achieve a greater like financial backing or financial power. In a shorter time. Whereas, I mean, you can look at like the ZK landscape right now, at least compared to some of the more splashy optimistic things that are going on. They're a little bit languishing, at least in terms of the narrative, uh, even though it's fully likely that in a couple of years or even sooner, they sort of end up being super important fundamental technologies. There is this interesting interplay between the narrative and the technology, but it's not clear that they're actually the same kinds of efforts in the first place at all. Yeah.

Ryan Smith: I think that's super insightful. And I think that, um, I think that only goes to show, um, the, like the benefits of like having like this just kind of random, messy, chaotic, free market where it's just like all of this non-repeatable stuff, just like it's happening. And it seems to be yielding, like it's like, it seems to be yielding really interesting outcomes, you know, like that you couldn't have planned maybe.

Nicholas: Yeah, absolutely. And, and then even as we're talking about this stuff, you know, so much of what you see on timeline, at least in certain parts of the world are really things that are more formed. Formatted like a company or presenting themselves like a company, but there's tons of stuff going on on chain that has a much more diffuse or original creative experimental approach that doesn't fit easily into that box and can't be so easily compared to, uh, to those things either. It also kind of reminds me, I was talking to somebody about Roblox the other day, which started off as a physics simulation for like high school science classes. And they observed the behavior of the students playing with the physics simulator after class and ended up making this game engine, uh, online game environment. Uh, subsequent to that. So I feel like there is a little bit of that in the Ethereum experimental personal bootstrap project part, part of the world too, where sometimes things may be end up becoming useful for a reason that wasn't so obvious at first, which you might not have as much time or bandwidth to consider if you're in a locked into a more like VC treadmill kind of approach.

Ryan Smith: Yeah, a hundred percent. And then, and then also, you know, again, like I think that the VC approach will also yield some fascinating things, right?

Nicholas: Like I, I, uh, Most of the time. Most of the things I use every day, I come out of that ecosystem.

Ryan Smith: Yeah. Yeah. So it's like, let's just make sure, like, I guess, you know, if, if I have to put on a hat, it's just like both the, both hat, let's just like, let's just do both.

Nicholas: Yeah, totally. So we've got a, through a great tour of everything that you're doing at index supply, shovel block API. Is there anything that we didn't discuss that you'd like to cover before we call it?

Ryan Smith: I mean, no, I feel like there's at this point, you know, it's like, sure. I can talk, you know, I can talk about a whole bunch of other stuff, but you know, I, I, I feel like this was a great, you know, kind of Intro to index supply.

Nicholas: Yeah.

Ryan Smith: Yeah. Intro to index supply. I feel like we were able to kind of like, hopefully now you kind of see like my, my own arc, you know, and like, cause you know, index supply is very much, you know, just like a, you know, a representation of who I am at this point. And, um, I hope it grows beyond and expands beyond that. But for now it's just like, it's me, like starting this arc, you know? And so hopefully, you know, I was able to, you know, share that with you and, uh, yeah. And that was. It was exciting for me to do. It was very fun for me to do. So I appreciate that. Oh, you're welcome.

Nicholas: Absolutely.

Ryan Smith: You got some, you got some, you got some very good questions and, uh, it's very been very enjoyable talking with you.

Nicholas: Well, thank you very much. Uh, yeah, I think it's a beautiful thing to see your journey from the start and also it's sort of evolution into this, uh, bootstrapped, um, I kind of contrarian approach to building infrastructure while everyone else is raising these millions and millions of dollars to do infrastructure companies, maybe building something, uh, a little bit slower, a little bit for the longterm. So if people want to check you out. There's index supply.com, uh, at index supply on Twitter and Farcaster and index supply on GitHub, anywhere else. People should look. That's it.

Ryan Smith: You got it. Awesome.

Nicholas: Well, if people are interested in building their own indexing solutions for their apps, uh, not many better places to go than index supply. And you can check out, there's a great demo of shovel and block API explanation on index supply.com. So I encourage people to check it out. Ryan. Thanks so much for coming through. Awesome.

Ryan Smith: I really appreciate the time.

Nicholas: Happy new year. Thank you to everyone who came to listen live. Thanks everybody for listening. Uh, in 2023, it's been a lot of fun. Happy to have this as a, the last episode of the year and one of the first ones to publish next year. Thanks again, Ryan. And thank you to everyone for coming to listen. See you around. Hey, thanks for listening to this episode of web three galaxy brain. to keep up with everything. Web three, follow me on Twitter at Nicholas with four leading ends. You can find links to the topics discussed on today's episode in the show notes. Podcast feed links are available at web three galaxy brain.com. Web three galaxy brain airs live most Friday afternoons at 5:00 PM. Eastern time, 2200 UTC on Twitter spaces. I look forward to seeing you there.