Web3 Galaxy Brain 🌌🧠

Nicholas: Welcome to Web3 Galaxy Brain. My name is Nicholas. Each week I sit down with some of the brightest people building Web3 to talk about what they're working on right now. My guests today are Bryce Ferguson and Jack Kearney, CEO and CTO of Turnkey. Turnkey is a private key as a service provider designed to let app devs and enterprise customers create, manage, and use private keys with a simple API. Turnkey leverages Amazon Nitro, a trusted execution environment as a service, to give customers of all sizes a secure enclave in the cloud. This is particularly useful for companies that need to programmatically manage a large volume of private keys. On this episode, Bryce and Jack explain Turnkey's Quorum OS, a deterministic OS for secure execution of apps like their Policy Engine, Signer, and EVM Parser. We also discuss the provisioning process for initializing apps securely and discuss the risk profile of hardware security modules and HSM clients. It was a pleasure chatting with Bryce and Jack and learning more about the wonderful world of cryptography in the cloud. I hope you enjoyed the show. As always, this show is provided as entertainment and does not constitute legal, financial, or tax advice or any form of endorsement or suggestion. Crypto has risks and you alone are responsible for doing your research and making your own decisions. Hey Bryce. Hey Jack. How's it going? Hey, how's it going?

Jack Kearney: It's going well. How are you?

Nicholas: Good. Thanks for coming on the show. I'm excited to have this conversation.

Bryce Ferguson: Yeah, same here. Thanks so much for having us. Absolutely.

Nicholas: I'm excited to learn more about Turnkey. You know, I guess I haven't seen you guys as much around on Twitter. So maybe before we jump into what Turnkey does, maybe you want to give a little snapshot of what your backgrounds are. We could start with Bryce and then go to you, Jack. Cool.

Bryce Ferguson: That sounds good. So to kick things off, my background is initially from Coinbase Custody. So Jack and I met while building Coinbase Custody. I was the first PM there. So really got to touch a bit of everything. I was working on product security. I was working on product strategy, new asset integrations, governance products, the first staking products at Coinbase. Got a pretty good lay of the land of how crypto was evolving, how key management was evolving. Near the end of my time at Coinbase, ended up kind of having the urge to do something zero to one again and jumped over to Trade Republic, which is kind of like Robinhood for your up over in Berlin. So was leading their crypto team and built the team from scratch, built the product from scratch. And launched a trading product all across Europe, which was really cool to see, but also kind of put me in that builder seat to integrate with a lot of key management providers, see what challenges developers were facing and see kind of a huge hole in the market around key management. So ended up getting back together with Jack last year to start Turnkey. And we both had been seeing the industry evolve in many of the same ways and start Turnkey.

Nicholas: Before we jump to Jack, I'm just curious, what like? who's the customer for Coinbase Custody? What kind of application does that have?

Bryce Ferguson: Yeah, it's a really good question. Coinbase Custody came around because Coinbase had obviously invested a ton in crypto security. And there are large financial institutions that have very specific regulatory requirements to hold their assets with what's called a qualified custodian. And that's basically a specific regulated entity that's able to hold assets on behalf of these registered investment advisors. So we were working with super large institutions, everything from grayscale to different crypto forward institutions like protocols that were holding their treasury and smaller kind of hedge funds and VCs.

Nicholas: Yeah. Jack, what's your background?

Jack Kearney: I've been in crypto actually for a really long time. First got into it in between my junior and senior year of college, summer of 2011. Wow. Kind of like very early days Bitcoin, still really, really nascent technology. Yeah. Crazy technology kind of just went down the rabbit hole basically with a lot of time on my hands as a student. Yeah. And then I kind of followed along, wasn't working in it professionally until 2016 when I joined Coinbase. Originally on the infrastructure team, but then was kind of the first person or one of the first engineers who went over and worked on Coinbase Custody. Which is where Bryce and I ultimately met. when we were, we kind of like, I guess, co-led the first staking project that Coinbase did, which was Taser staking. And then did that for a while. So to kind of just, you know, got Coinbase Custody zero to one, and then I peeled off and went over to Polychain.

Nicholas: So I was early.

Jack Kearney: Polychain, I guess. Did just a lot of like custody and staking work at Polychain too. So we had built out like a number of, you know, internal kind of costs. Custody systems. at Polychain was operating a number of the pretty large like staking operations for Polychain's assets under a sort of side company, which was called Polychain Labs. And then basically through that process, just saw kind of this gap in the market where I was like, I want to do more stuff on chain, but in a controlled way. And just felt like there wasn't like the product that could really do what I wanted to do. And so Bryce and I got back together and we were like kind of daydreaming. About what we can do. And ultimately Turnkey was born out of that.

Nicholas: Yeah, it's the perfect transition. So what is the problem that Turnkey is trying to solve?

Bryce Ferguson: So Turnkey really noticed that all of the key management providers in this space were geared towards this financial institution, qualified custody use case. And most financial institutions at the time were looking to just buy and hold crypto. They weren't really doing much with it, not participating on chain, not building applications on top of that infrastructure. Really just kind of locking it up and throwing away. And so with Turnkey, we wanted to be super low level infrastructure geared towards developers and geared towards people who were either doing massive on chain automation. So the likes of the polychains of the world that have massive staking operations or smart contract deployments, things like that, or people who are building applications. So folks that needed wallet infrastructure, folks that needed ways to embed crypto functionality directly. And so we wanted to take this infrastructure approach to modernize key management and give developers much more flexible tools around keys.

Nicholas: So would you say that programmatic private keys infrastructure, custody infrastructure is kind of what Turnkey works on?

Bryce Ferguson: Yeah, that's exactly right. And I think it can kind of be a hard discussion sometimes because as we've started so low level, we're almost a layer below all the other players in the space. Turnkey can be applied in a lot of different ways. So that can be programmatic, you know, fund operations. It can be wallet management. It can be, you know, a key management backend for another developer tool. There's a lot of different directions and a lot of different kind of use cases that we've seen over the time that we've been live.

Nicholas: So are the ideal customer profiles people like wallet SDK providers or people like polychain or straight into app developers as well?

Bryce Ferguson: Yeah, it's mixed. I would say the two. The two big use cases that we've seen is everything around asset operations, and those are typically larger crypto organizations that are either doing staking, treasury management, smart contract deployment. And then the second bucket is everything around non-custodial wallets, and those could be applications that are directly integrating with with Turnkey like Goldfinch that has a really great demo of kind of a passkey wallet that they put out into the space. Or other developer tools like Dynamic and ZeroDev that have integrated with Turnkey as a key management backend or their wallet as a service product.

Nicholas: Is that the DeFi for the world dot com demo you're referring to?

Bryce Ferguson: Yeah, that's exactly right.

Nicholas: So that one's basically like a passkey wallet, right? Or passkey, I guess, passkey wallet into a. is it a smart account that it's attached to? I don't remember what's on the other end.

Bryce Ferguson: Yeah, that's right. It was pretty cool. They used both us and ZeroDev to have passkey powered smart contracts.

Nicholas: So basically, you go to the website DeFi for the world dot com, and it'll generate a passkey, I guess, using the Turnkey API. Yep. And so I guess let's let's start to jump into what actually is Turnkey. But even just thinking about that scenario, like when I think of a passkey, I think of it as a web API that's generating a passkey on the secure enclave or a trusted execution environment on the device. So why do I need an API?

Bryce Ferguson: Yeah, it kind of gets into our security approach, which I think would be good. But at the highest level, what we've really done is we've abstracted away the private key to authentication. So we are allowing folks to access a private key in a cloud based secure enclave via that passkey and sign with that.

Nicholas: I see. So the so the private key lives in the cloud, but is and I guess we'll get into the details, but is access restricted to the one user and it's not just a private key sitting on a server somewhere. That's right.

Bryce Ferguson: Yeah. So we have this set up, I think, a really great metaphor of the security setup is it's kind of like a safety deposit box vault where Turnkey is securing that overall vault, but only the user with their passkey is able to access the individual safety deposit box where their private key is held.

Nicholas: I see. Okay. So I guess, yeah, Jack, maybe do you want to dive into like kind of what is Turnkey offering? if I sign up today and start using it as, let's say, an app dev is maybe the most approachable example. How do I start to integrate it? What do I get and how does it work inside?

Jack Kearney: Yeah. So the hope is that it feels a lot like interacting with any other normal API that you might might interact with with kind of one big caveat, which is which is how we do authentication. So all authentication to Turnkey's APIs is done using asymmetric cryptography. So there's no like passwords anywhere, no traditional like API bearer tokens anywhere. We do all kind of client side signing of every every request that will ultimately make its way into Turnkey's backend and and and like be signed by a key that that theoretically is held in Turnkey secure enclaves.

Nicholas: So that's WebAuthn?

Jack Kearney: So, yeah, we use WebAuthn. We use for our like desktop kind of web application. We use passkeys. You can also sign with just API keys that are like a public private key pair where we know about the public key in the backend. You have the private key is held on the client somewhere, could be in a terminal, could be, you know, it could be anywhere. Really, you could do like air gaps communications where the private key is on an air gap machine and you sign a transaction or like an activity for for Turnkey's API. And then we just ingest it in our backend and and take action on that.

Nicholas: Right. So I guess to spell it out for people who are sort of new to passkeys, I guess the unique property of pass keys in the WebAuthn approach there would be that even the user themselves. Right. is never shown the private key. it's just generated in the secure enclave and then the public key is given to turnkey to use for authentication. but the os or the web browser manages access to that to that private key and the user never. it's never in memory in the application or the web app.

Jack Kearney: yeah exactly exactly.

Nicholas: you're saying you know. you can also just bring your own private key if you want to just generate it. you know some traditional way.

Jack Kearney: yeah yeah if you wanted to just you could generate one using our cli or um our sdk or you know in any other way.

Nicholas: okay so you come up with some uh public key uh associated with a private key of whichever type and that's how i'm going to authenticate. uh that's how a user is going to authenticate when they try to access their turnkey. i guess safe deposit boxes maybe the metaphor.

Jack Kearney: yeah exactly and you know. hopefully this is all abstracted away and it's like a clean interface for a developer. so we don't want it to feel like any different than if you were just interacting with. you know any other api you might interact with in traditional web development um but i think it achieves basically some really interesting security properties for virtual key. um overall. so namely that we basically have like two classifications of services in our in our back end those that are super sensitive let's say so things that are decrypting private keys or evaluating policies associated with your account or um you know notarizing data things like that. and then services that aren't super sensitive. and without fail the super sensitive services are deployed into um secure enclaves and the services that aren't super sensitive are not deployed in secure enclaves. and ultimately then a signature happens on the client side is ingested into our api that like intermediate layer of our api is actually like unable to do anything. it's not really permission to do anything. theoretically if like an attacker were to compromise that intermediate layer they wouldn't be able to take action on behalf of the user's account. if like you know you're not able to uh turnkey engineer were to accidentally get an image that was doing something um you know not unexpected into that intermediate layer we we wouldn't be able to like take action basically do and do anything really with the user's account. the only thing that could ultimately result in the enclave performing an action on behalf of the user's account is an authenticator which is which is only held on the client.

Nicholas: so basically you're saying that the execution of code uh with the private key that's in this secure uh turnkey uh environment which we'll talk about in a second in a little more detail can only be executed if with a signed message from the passkey from the secure enclave on the user's device. is that right? yeah yeah exactly so it's a little. it's maybe a little confusing for listeners because we're using the term secure enclave to describe both compute that's happening on the client and for authentication and on the server for accessing the pk. that's there. i guess part of the first question that occurs to me here is like why do i need a separate thing that i'm authenticating into if i already have a pk on my device?

Jack Kearney: yeah so let's let's talk about i guess ethereum or like bvm specifically you can't efficiently verify signatures that are signed by um you know a passkey or or um you know most other kind of secure enclaves on the client side in the ethereum blockchain. um there are things like eip 7212 um which is a pre-compile for for that verification. but right now if you were to try and set up like a smart contract wallet where you were having a passkey private key as as like one of the authenticators for that smart contract wallet it would be just extremely extremely expensive. so using this structure we're able to authenticate to turnkey's back end which kind of pivots to a private key that is a traditional ethereum eo key um and ultimately that's just like a much more efficient way to um you know authorize basically

Nicholas: transactions

Jack Kearney: and then you can just like do transactions using pass keys on evm chains

Nicholas: i see. so for people who are interested in that they can go check out the other episodes of the show about eip 7212 and a few other episodes. if you search on the website for pass keys you'll see or 7212 you'll see other detailed information about that subject. but basically the pass keys are this sccp 256 r1 curve which is not the ethereum and bitcoin curve so you can't directly sign. so you're basically generating a k1 curve inside of an hsm and then authenticating to get access to sign things with that separate private key um via this turnkey mechanism. exactly yep okay very interesting. so maybe i was reading some of the documentation and i i saw this quorum os. uh maybe you could describe a little bit about like how the thing actually works. what's going on under the hood?

Jack Kearney: yeah yeah yeah um. so yeah we have a number of of these enclaves actually deployed so it's not it's not like just one single hsm or secure enclave. that's um responsible for like performing all actions um on behalf of a user or in response to users requests. instead it's kind of like a traditional microservice architecture. um but where like all the very sensitive services are deployed into these secure enclaves? um and so we we decided we were going to like come up with the like. the deployment process essentially for all these different services was going to be the same and so we wanted to standardize how that was going to work and we wanted to make sure that when we set up these super secure services we had a very predictable way of standing them up. um we could essentially you know verify prior to provisioning them with a very core secret that's you know capable of doing something in the turnkey backend wanted the people who are kind of provisioning those services to be able to to verify that. you know they were provisioning the right service. they weren't provisioning kind of a malicious service or um you know a service that hadn't sort of met all of our um verifications and stuff. um so so that's where quorum os was born.

Nicholas: i'm going to talk about having like. when you think about them as microservice why is it not just some monolithic process. why is there a need for multiple separate processes? uh to segregate the risk or sandbox different customers or something?

Jack Kearney: no it's yeah it's more. just like scalability and um ease of like. you can more easily scale up individual components of the system. um you can more easily just like say we wanted to just rotate. um like we've done a lot of development on one of the services it wouldn't require us like redeploying entire monolith. we can just deploy that one thing. so it's just kind of like you know general best practices from running large infrastructures in the past.

Bryce Ferguson: i think one thing i'll add to this as well is as we were thinking about this architecture we realized that a big shortcoming of a lot of the folks in key management is they were focused just on the security of the private key but not all the other aspects of custody and the security of that. so every single one of these services is these different sensitive aspects of custody whether it's transaction parsing or key generation or signing or the policy evaluation and we do those in such a way that we can just have a very holistic security model that that takes into account all these different

Jack Kearney: aspects

Nicholas: i see and so i'm starting to see why. you know talking of scalability and stuff thinking about it maybe in that polychain example for staking operations where you might need to spin up lots of different private keys uh in order to manage some massive operation it's not just a single user authenticating on their phone corresponding to some smart account but maybe a really sophisticated programmatic application of crypto accounts.

Jack Kearney: yeah absolutely like we. we wanted low latency um you know and and high throughput basically on this api. for is it ultimately like if you know if we're powering like a number of large consumer applications using this passkey approach like it could be? you could imagine it would be like a lot of traffic api um and so we wanted to you know from the ground up. basically imagine um you know just dealing with with a lot of load.

Nicholas: basically i want to continue on the quorum os and and more the technical stuff but just before that does does this have applications like completely outside of blockchains?

Jack Kearney: uh yeah we i think it definitely could. i mean we're we're we're focused on crypto and blockchains and stuff. right now just because you know we're an early stage startup you have to find focus and um and fit somewhere. but i think over time i think definitely applications we could find outside of crypto for sure all right.

Nicholas: so sorry i interrupted you with your technical explanation. so yeah take us through it yeah.

Jack Kearney: so so that's where quorum os was born really um we wanted a very predictable way to stand up and reconstruct a core secret inside of a secure enclave um. and so quorum os is just a super super stripped down operating system. um there's there's no tcp isp stack in it basically just a kernel. and then this like custom rust and knit process we've developed. and really what happens is you deploy quorum os you deploy it with you sort of configure it with an application that will eventually be run and then people get around and um can go into more details on kind of the provisioning process later. but we use shimmy secret sharing to essentially reconstruct a core secret inside of that enclave. so individual people have uh portions of this secret. um the secret is reconstructed inside of the enclave. um it is encrypted i guess in flight all the way down um. and then after the the core secret has been reconstructed the application that the enclave was actually deployed with is essentially just exact. so you kind of just pivot to this application and suddenly the application is running it has access to a core secret and then

Bryce Ferguson: um

Jack Kearney: you know it can perform actions in our back end and those actions can be like um you know decrypt a key that is uh relevant for a particular user or um sign a message that another enclave can verify and you know confirm that. oh yes this was like. we have as bryce mentioned before we have a policy engine. that policy engine is um you know before we'll sign anything the policy engine has to actually like authorize the signature um and so the policy engine would basically sign off and say yes i've verified that this um this request conforms to the policies of this turnkey account and so please go ahead and sign using this private key that is associated with that turnkey key account.

Nicholas: what what is a is a core secret just a private key with certain permissions associated with it.

Jack Kearney: it's it's actually it's. it's just a secret that then you can derive other keys from. basically so it's it's a little more general than just a single private key because in some cases we want to drive like an encryption key sometimes we want to drive a signing key. um it's just kind of like a a secret that we can use to um perform any number of kind of other cryptographic operations with okay and then the the keys that are derived from it are associated with specific applications exactly yep.

Nicholas: and so what's an example of an application?

Jack Kearney: yeah so the policy engine which i was just talking about um is a great example. so we have um a custom language. you can write uh policies that basically dictate who and under what circumstances. um can like sign using a key held in another application um which we call it the signer um. and so yeah policy engine is one signer is another one. um we have kind of another input to the policy engine which is like medium parsing um application. so we can like take an ethereum transaction or any evm transaction and parse it and be like what is this transaction doing and then allow people to craft policies that um sort of like take. you know human readable basically um aspects of of an ethereum transaction. so it's so we can say like okay this you can sign an ethereum transaction if it is you know not spending a crazy amount of gas or something like that.

Nicholas: so i guess if we go like in the order of operations and just walk through setting up one of these things the first thing would be to create a core secret and associate it with a application. is that right?

Jack Kearney: uh yep yeah so we did. when we were like first getting turnkey off the ground we generated a number of core secrets. um those were all generated inside a secure enclave that we there was never a moment where they were like exposed to. um you know any single machine that wasn't verifiable um. and then uh we split it. so you basically that we called it the genesis application. basically the genesis application was configured with a number of public keys which are associated with um. we call them quorum os operators. so people who are basically responsible for deploying and managing quorum os instances and those operators have uh essentially yuba keys which have a private key on them. and so that's listen there's a public key associated with the private key that's held on the yuba key. sorry this gets very confusing. i wish we had like a whiteboard in front of us. um um so we uh said the genesis application runs and it's provisioned with the public keys that are associated to the yuba keys which are held by the quorum os operators.

Nicholas: so basically you have to hold you have to have is it shamir over those yuba keys?

Jack Kearney: so it is. so we generate the core secret in the genesis application. it is then shamir'd to the and then re-encrypted to the public keys which are held on the yuba keys.

Nicholas: okay and so holding um a sufficient threshold of the yuba keys allows you to do what to.

Jack Kearney: yeah so holding a sufficient threshold the yuba keys would allow you to just reconstruct this core secret which we'd never do outside of quorum os. you know the process then to actually reconstruct and the whole thing like. we have a crazy process using production laptops and um which we can get into in a second. but yeah it's a very rigorous process to essentially then take the yuba keys and decipher text that those yuba keys can decrypt um and actually transport it up into the enclave after the fact.

Nicholas: so with once you have the threshold of yuba keys you're able to reconstruct the core secret inside of. uh and that's the genesis core secret or maybe some derivation of it.

Jack Kearney: yes so we application that we called the genesis application. okay um but then after it's generated you essentially associate it to.

Nicholas: you know another application that we performs a specific task in our back end and then you would use the yuba keys only when you're trying to set up a new application.

Jack Kearney: uh no every time we redeploy you're using you're using those yuba keys okay. so when you're making updates to quorum os essentially yep when you're making updates uh not actually to quorum os but to the applications that are run with quorum os okay.

Nicholas: so very example you update the policy engine.

Jack Kearney: yep yeah you change the language. the policy engine is um operating on. we need to redeploy it and that requires us breaking out the yuba keys breaking those production laptops and going through this very very rigorous process to reconstruct the the same quorum key inside of an enclave running the new version of the application.

Nicholas: i love it. it's like i don't know 24 or like evangelion or something. it has a. it's very cool. uh is there a media property that you guys think about when you? uh when you do this?

Jack Kearney: no uh we should definitely find.

Nicholas: yeah we need a social network for whatever you call this business. um so okay so you can then reconstitute the. but even when the core secret is reconstructed it's only happening inside of a secure enclave that's on amazon servers or something like that. uh yep exactly so even though it's being reconstructed it's being reconstructed inside of a process that if you were to observe the process from the outside you cannot see the private key or the core secret itself.

Jack Kearney: yeah exactly so we use this product called aws nitro enclaves. um which is? it's it's sort of amazon's flavor on a a standard that's called the tpm 2.0 standard for secure enclaves. um so it's not quite tpm 2.0 but it is pretty similar to it. um and basically you spin up a vm that vm you like cleave off a couple like some of the cpu and memory and assign it to this enclave and then the original vm like the remaining cpu and memory is just there's. there's no like link between them. you can't like view each other's memory at all. um all there is is basically a essentially a virtual serial cable connecting the two. and then um yeah that's that's kind of how nitro enclaves work at a very high level.

Nicholas: and nitro enclaves and tpm 2.0 these are all vms or there's actual hardware like hsm is about hardware.

Jack Kearney: right yeah so tpm this gets a little confusing. but in amazon's in nitro enclaves it is um yeah there's. there is a physical piece of hardware. it's like a a pci card basically attached to the physical machine is my understanding and they're a little cagey about this. but that's my understanding. in the tpm 2.0 standard it is um there's an actual like physical separate device which is it's. it's typically called a secure co-processor it's kind of the generic word for it and sort of sits next to the secure enclave and watches this year secure enclave as it boots up and and takes measurements about the secure enclave um. so it says like okay i see this thing boot up um and i saw a kernel get loaded and that kernel has this hash and then um you know there's uh kind of the the boot disk had this hash and and stuff. so it takes these like pretty objective measurements about what it observed when the enclave was booted and then it's booting up um which ultimately then allows us to to basically verify that yeah okay we have this independent

Nicholas: kind of

Jack Kearney: thing that we didn't even control because we don't control the tpm firmware or in amazon's case the nitro security module firmware which is basically their equivalent of a tpm. um. and uh so we have kind of this independent third party. it's like watchdog basically and we can say is this running like the version of quorum west that we think it should be running and if it is then we can go ahead and provision it. if it's not we can call up our account reps at aws and say what's going on.

Nicholas: so quorum os runs inside of the hsm or tpm or amazon nitro instance itself.

Jack Kearney: yep exactly interesting.

Nicholas: so i assume the listener like me is not super familiar with all this stuff. so basically you've written software that runs inside of a secure enclave on amazon infrastructure and you're writing now software on top of that what you're calling quorum os to allow people to deploy specific applications. but most likely i guess they're going to be using applications or maybe it's above quorum os is the idea that people would deploy their own applications. or primarily it's going to be things like the policy engine the signer and the evm parsing app that are your creations that other people will just make leverage for their applications.

Jack Kearney: yeah so so right now it's just our applications. um long term i can definitely see a world where we expose this and allow people to upload their own applications basically to run inside of this. and actually we we are hoping to open source quorum os before too long so that you know even if even if it's in our infrastructure or your infrastructure you'd still be able to write an application to run inside of a secure enclave.

Nicholas: okay so so let's keep going through the kind of chronological process. so you come up with some generate some core secrets uh or a core secret and derivations of it. you use the you kind of deploy quorum os. uh with this shamir yubi key setup that allows you to re or redeploy the applications on top of quorum os. should there be revisions to those uh applications? that roughly okay summary yep so well.

Jack Kearney: so you first run quorum os okay and you're running it with an application um but but it is not yet provisioned with that that secret. and then we go through a very rigorous process using um dedicated basically what we call production laptops to re-encrypt. so what happens when quorum os boots up? it generates an ephemeral um key pair. so it basically says i i'm running quorum os first time i'm going to just generate a new key pair and then when a user then requests what's called the attestation document from the nsm or tpm quorum os says okay i'm going to put this public key into um for the ephemeral key pair into the attestation document. the attestation document includes um the measurements that we were talking about before. so the measurements of like what kernel is running there and other kind of objective properties about the os and then the quorum os operators see the attestation document they say okay this is running the right version of the kernel is running the right you know has other properties that i expected to have. i'm good for. i'm like. i think it's like a valid version of quorum os. i think that valid version of quorum os is also going to run the right version of the application that we want to run. and so i'm going to re-encrypt the core secret I have to the ephemeral public key that was generated when that Cormas instance booted up. And then they transport that ciphertext basically over to that enclave. The enclave is the only enclave that then can decrypt that ciphertext because it was the one that generated that ephemeral key there. And then a number of people do that. And we have a threshold number of those. So you sort of hit the Shamir threshold. And at that point, you can reconstruct it. You pivot the core secret and then you pivot to whatever application we're running.

Nicholas: So essentially that's installing an application, installing CormOS and then installing an application in that instance of CormOS. Yep, exactly. And when we're talking about applications again, we're talking about things like the policy engine, signer and EVM parsing app. Yep. And so the reason that's all important, I guess you would have one, or maybe if the customer base is scaling, you might have multiple instances of say the policy engine. But the reason all this, this is so important is so that people who are users of, you know, devs who are users of Turnkey can trust that the policy engine, for example, is really doing what the, I guess, eventually open source code says it should do.

Jack Kearney: Yeah, exactly. So eventually we hope to open source CormOS and maybe a little further down the road, we want to open source, you know, the signer and the other, the policy engine and other applications we have. And the idea is that you can look at the source code, as a developer, and you can be like, okay, I see what they're doing here. That looks safe. That looks reasonable. You can compile it yourself. And then you can basically compare the artifacts that you got when you compiled CormOS or you got when you compiled the application against what is running, what is actually running. And you can confirm that like, okay, the source code I reviewed is actually the source code that was compiled down as ultimately then running in the enclave.

Nicholas: It's interesting from a philosophical perspective, because it kind of is offering some of the same properties that something like Ethereum is offering in terms of knowing that the code is running, but in a very different way.

Jack Kearney: Yeah, in a different way. Currently it's centralized in a pretty meaningful way, but like, yeah, we, I mean, it's kind of funny. Like sometimes, you know, like from one point of view, you look at turnkey and you're like, this just looks like a normal, like API. This looks like a normal Web2 API. And then you like shift it around a little bit and you look at it from a different angle and you're like, wait, this looks a lot, This looks like a decentralized thing from the core.

Nicholas: Bryce, what were you going to say?

Bryce Ferguson: I was just going to chime in that. another thing that we're trying to do with CoreOS is just remove single points of failure. So a big piece of this is no single turnkey engineer is ever able to deploy code to these enclaves. And we've kind of taken that philosophy throughout our entire stack.

Nicholas: So that, and that's implemented, for example, by this YubiKey process.

Jack Kearney: Yeah, exactly. So you don't have to trust. And so actually, like, there's another step that happens before you provision the, you know, CoreOS instance with your secret, which is where you would actually build the application yourself. So we'd say like, okay, we're deploying this tag of a signer, like a get tag. And then the people who are going to actually provision it would literally compile it on their production machine and be like, oh yeah, I got the right version after I compiled it. And then confirm that that version is in fact running inside of CoreOS. Go ahead and provision it.

Nicholas: So you have a threshold of people who have independently compiled.

Jack Kearney: Yes, exactly.

Nicholas: Very cool.

Jack Kearney: This is like one of the, like a very hard thing we do. I don't want to say it's the hardest thing because there's a lot of hard things at turnkey, but like typically, like a lot of software is actually not what's called deterministic or reproducible when you build it. So oftentimes there's things like timestamps or, you know, other things that actually get ingested in the compilation process and result in a different artifact, like a slightly different artifact being produced when if you were to like compile it on your machine and I were to compile it on my machine. But we've gone through this very laborious process of ensuring that every single artifact in this, like CoreOS and all the applications on top of it, they are deterministic. And so we can basically confirm that. We can, you know, I can compile it, you can compile it, and we'll get the same artifact byte for byte resulting in the same hash byte for byte. And then we can confirm that. You know, it is running what we think should be running.

Nicholas: Yeah, I was very curious about this idea of a deterministic OS that you described in the docs for Quorum OS. Are there other deterministic OSs that people might have heard of?

Jack Kearney: I don't. actually, I don't think they're, like all the way down, I don't think there's anything that's gone quite as far as we have there. There are certainly like you can, there are ways to do like, like if you have a Go application or Rust application, there's ways to, you know, produce a deterministic artifact on just that language. But like we were really concerned with going like all the way down, like as down to the core as possible. And have some folks inside the company who just live and breathe this stuff and you know, have gone through a pretty crazy process to make this all happen. Another thing we're actually hoping to open source here pretty soon is, it's essentially like a package manager. that is where like all the artifacts are, are fully deterministic. And so we're hoping to basically produce just OCI images, so basically Docker images for, you know, various common tools that people might want to use or build containers for like Go that you might want to build deterministic artifacts with. And yeah, so hopefully gonna release that sometime in Q1, I think.

Nicholas: - And those would be dependencies for applications that run on Quorum OS?

Jack Kearney: - Yeah. And in our case it would be, but you could use it for, if you wanted to do something totally separate, you could use the deterministic package manager to produce an artifact that was deterministic.

Nicholas: - And the essence of a deterministic OS or process piece of software is that it doesn't introduce different data depending on when you run it.

Jack Kearney: - Yeah, so every time you compile it, you get the same thing. So like, 'cause this is like a, I think this is, a lot of people, leverage secure enclosures and they say, okay, yup, we're deploying our artifact and secure enclaves. So that's good. We've isolated the workload and you know, that's, that's cool from one level. Like you, yeah, you have isolated the running application from other servers that you have in your cloud, or, you know, potentially developers who are, have SSH access or something to a machine up there. But the problem is oftentimes in a lot of, you know, tech companies and all, infrastructures there's just like one machine that's responsible for or one process that's responsible for building all the artifacts. and so even though you've like done all this probably difficult infrastructure work to get your secure enclave like deployment set up you you still have this kind of lingering single point of failure in the build process. and so we said we don't want that single point of failure we want to remove you know any single machine that's not an enclave from the critical path from being able to do something super scary in our back end. and so we said well we kind of have to make everything deterministic. if if everything is deterministic then we can still have a ci process we can still have a machine that's building the artifact before it gets deployed into the enclave. but we can then people can confirm that what was produced by the ci is actually like the artifact that should have been produced by just literally rebuilding it themselves.

Nicholas: so the deterministic uh aspect of quorum os and of the software you're writing is enabling. it's both removing a central point of failure dependency on a single process that does the building of all the software. and it also allows you to do this ubq provisioning thing.

Jack Kearney: yeah it allows you to to before doing the ubq provisioning thing. it allows you to confirm that what is running there is what you think should be running there.

Nicholas: it allows you to do the parallel compilation on the production machines.

Jack Kearney: yeah and the and i think i should i don't think i've said this phrase yet but the um the process of of like verifying the attestation document that's produced from the nsm in our case or a tpm more generally. that's called remote attestation. you'll you'll see that you know thrown around as a kind of word to describe this.

Nicholas: wow okay so this is very deep. so okay. so we've gotten to the part where i think now we've got an application running on a remotely attested application running on a quorum os instance in an hsm right mm-hmm. so now we have things like the policy engine and the signer and the evm parsing app. so what does the policy engine for example do and how do i interact with it?

Jack Kearney: yeah so okay. so we have built a custom language which is essentially allows you to express the conditions and the people or the authenticators. i guess more generally that should be able to take an action on behalf of your turnkey account or we call it an organization but it's essentially an account and so you can say okay this there's this private key. it's held in turnkey and before turnkey signs using that private key any transaction we need to have um you know bryce and jack both need to sign off on that action and then the policy engine would essentially validate that the um. all of that has happened and then if if all that has happened so it observed that both bryce and i signed a a a transaction it would then sign a message to the signer to say okay yes you should go ahead and sign this because i've verified the policies that were in place for this turnkey organization have been met and so according to you know how this this organization was set up. that means we should go ahead and sign using the key that is held in turnkey.

Nicholas: and where do the policies? where does that state live?

Jack Kearney: inside the hsm itself that state so when it is evaluated is held inside of the um inside of the secure enclave. yeah when it is like at rest it is not held inside the secure enclave so it's held outside and just a pretty traditional database.

Nicholas: how is the policy engine able to know which policies it must validate for a given transaction?

Jack Kearney: signature request yeah so you sign. so basically you you've gone ahead and you said i want to sign this request. i have a yuba key on my machine um or a web auth and credential of some kind. you sign a payload. so actually this is kind of cool. um as a quick tangent here. so we basically leverage um this field in the web auth spec which is typically reserved for just a challenge. so usually the way it goes the server would generate a challenge and then the client using web auth and would would sign that challenge using using their credential um and then kind of send it back up to the server and the server would say like oh yeah this i see that the credential signed the challenge and so i know that the public key that's associated with the the private key and the web auth and situated that user actually has it. and we kind of like abuse that challenge field. so we say prior to signing a payload you're going to take that whole payload you're going to hash it and then you're going to basically tuck that in the challenge field and then sign it. and so by the time it makes its way all the way to the policy engine you've included the full payload and then you've included this signature over the entire payload and then you've included the full payload over the payload and then we can say okay we see we know this. this um like public key basically is associated with this um turnkey account. this organization and we're gonna verify that that public key which has previously been known to be associated with this turnkey organization the private key actually did sign the request and then after it's it's uh it's signed. the policy engine would be like okay i see it signed um using the credential i previously knew. if it conforms to the policies that were already set up in the organization i'm going to sign the authorization to then go ahead and sign using the private key that's also held in the organization.

Nicholas: that's cool. so basically you've compressed it from an authentication step and then uh signing request step to just one step like basically by authenticating with the web authent passkey or ubkey or whatever. they're able to directly sign the payload request instead of authenticating first and then being presented to a separate challenge related to the uh yeah the transaction they were being asked to request.

Jack Kearney: interesting yeah you could. i mean ultimately that's just kind of up to the application developer like how they want to leverage that. but but yeah you that's a really cool aspect of this is it can feel a lot like you're just directly signing ethereum transaction using.

Nicholas: you're using web authentic i guess because depending on how let's say it's a web authentic approach you might have to let's say it's on a phone. you might have to face id each time you want to access the passkey. uh and this way it could be just one one face id request.

Jack Kearney: yeah well yes i mean currently. currently it would have to be multiple face id requests. but theoretically at some point in the future we could build a. we could build a feature to allow you to like sign multiple concurrent like crypto transactions using a single face id basically just by by using some relevant data as the challenge instead of yeah exactly random data. yeah. so if we wanted to implement that it'd be like we have an array of a bunch of different hash that whole array of transactions and then sign that and then in the back end we can interpret that as like oh we should then go ahead and sign each one of these individually using the crypto private key.

Nicholas: that's extremely cool. how long have you been working on this as a team?

Jack Kearney: well i think we. i think first conversation i had around turnkey was almost exactly two years ago. so it's kind of december 2021 and then we we actually raised money in march 2022 and then we were there was like bryce was still wrapping up at trade republic and um just kind of like doing some other sort of like administrative work to get it off the ground like incorporating and stuff and then actually only started really really building in earnest in may 2022.

Nicholas: was that a good time to fundraise for such a theme?

Jack Kearney: it was a great time to fundraise in march of 2022 and we we squeezed in the window by like a month and a half.

Nicholas: nicely done nicely done yeah um okay. so have a policy engine. we talked about. um i'm curious about the evm parsing app. so the idea is that one of the applications that maybe is less. uh secure super sensitive security?

Jack Kearney: no definitely high security high security yeah high security yeah. so that you know basically ingests ethereum transactions and all it does is parse that and basically parse it into like json um and then the policy language we have you can specifically reference the the fields so you can be like the ethereum transactions. to field has to be this address or ethereum transactions. you know gas field has to be less than this amount or things like that.

Nicholas: okay interesting so you can basically do permissions over. i guess it sounds like a little bit overlapping with policy but instead it's it's about the the state that you're the the transaction that you're going to sign or the the contracts you're interacting with.

Jack Kearney: yeah it's what it's like.

Nicholas: ah okay i see and how does the signer fit into everything?

Jack Kearney: so the signer then is is kind of where the signer um going back to the the core secret as we were calling it before. the signer actually has um a number of keys that are derived from this core secret. um one is an encryption key. so the signer is the only thing that can actually encrypt the or sorry decrypt the private keys which are held in a database outside of secure enclaves um and then it also has a signing key. so it can you know notify other enclaves. that has done a thing. but essentially then it just decrypts private key um or like i guess the ciphertext is pulled into the signer signer decrypts that private key and then if it has been authorized to be signed by the policy engine it will go ahead and sign using the private key.

Nicholas: wow okay this is uh it's very deep. uh subject i'm i'm very impressed by the kind of. i mean it's not just a little sugar on top it's it's pretty deep. but you know we talked about kind of like what the purpose of all this is and obviously i can imagine someone like polychain labs needing this kind of programmatic access to private keys and not wanting them to live on any one device or process in the cloud that they have to administer. that seems very clear to me. is the like uh end user application is it primarily about the r1 curve not being in the evm? or is there something else for the future of wallets that makes sense here?

Jack Kearney: i mean i think we we just envision this as like a really good primitive to all sorts of things is i think the right way to think about it. so you might you might imagine building um you know a wallet. yeah that's that's that's a real you know i think awesome use case like we've. um we've done a lot of work like bryce is talking about the beginning with like goldfinch to demo how you might do this. and um you know then you could have a situation where goldfinch can write this application that you know interacts with a series of smart contracts but really like in a very provable very rigorous way. put the authority for put the authority to actually sign um and interact with those smart contracts on the actual user. um. so we think there's just you know a ton of different ways this could ultimately be um assembled and kind of different applications that could leverage this i think.

Bryce Ferguson: another thing i'll add there is we're noticing that people want to have a variety of authentication and recovery primitives that they can pick and choose from as they're designing their wallet experience. in general wallets just aren't going to be one-size-fits-all. we're having people who are building much more institutionally oriented wallets where you have multiple business users who are accessing one single underlying wallet. you have people who are making gaming wallets for example where there are different security and authentication considerations. and so although we're really bullish on pass keys and we think that authentication is going to go towards pass keys in the long term we want to offer a whole bunch of different ways that people can plug and play different authentication methods user experiences so on and so

Jack Kearney: forth

Bryce Ferguson: into types of wallets.

Nicholas: uh over time that makes sense. you mentioned recovery and maybe well i guess did jack did we finish the kind of how to use it? uh timeline of interaction or is there anything? did we not really get to the end? is there anything that we should finish up?

Jack Kearney: i think maybe the only like. other thing i'd say is that because core mos was written and uh you know generic kind of modular way like we can really run arbitrary code we can write any kind of like application we want in crisis. well we're actually one of the like one of the things that i think is really cool which we're starting to think about now and this actually kind of goes back to what bryce was talking about. i'm not actually starting to think about it. we're pretty far in the process is we're? we're gonna do um essentially a lot. authorization to the secure where like the actual authentication the tls verification is performed in the enclave. so it's it's ending up being this like very interesting like provable way to ingest data into a very secure system. um it was. i had a friend reach out to me the other day and he's he's building a crypto network. and was he basically wanted to ingest um output from chat gpt in a verifiable way like provable way into his his crypto network? and and i was like wow this is actually like a pretty reasonable use case and we could we could probably do that without a ton of work. um to i mean it would still be on on open ai side. it'd be like central party and stuff. so i'm not saying that like. the output of from their side would be um you know necessarily as as like rigorous or or the security wouldn't be like quite as bundled up probably as what's happening on our side. but um the idea of of just ingesting arbitrary data into the system i think is a really cool kind of like you know addition.

Nicholas: um so you could do something like multi-factor of like. you could require someone log in with their meta google twitter accounts and then those services would authenticate past the proof out of jwt whatever bearer tokens however it works and then that could be interpreted and validated inside the secure enclave. so you know that you can't. you know you're trusting google that Google's returning the right result, but the validation that multiple of them, for example, have been successfully authenticated by the user could be executed inside of the secure enclave itself, inside Quorum OS.

Jack Kearney: Yeah, exactly. So you'd have to trust Google there, that Google had your identity and Google did their leg of the OS process correctly. But you kind of wouldn't have to trust Turnkey, and so far as you wouldn't have to trust Turnkey's intermediate API. You'd have to trust Turnkey's secure enclaves, but assuming that you verified the code of our secure enclaves, you could go as deep down as you want and be like, "Yeah, they definitely confirmed that correctly.". And so, yeah, I'm trusting Turnkey.

Nicholas: Right. I guess that is one question that's on my mind. Is there no Intel management engine for Amazon Nitro HSMs, or why should we trust this?

Jack Kearney: Well, why should we? So yeah, the trust factors here are a really interesting avenue to explore, because a lot of the security comes down to trusting the objective measurements from, in this case, the NSM, and more generally, in the TPM. And that's basically... So essentially, the attestation document we were talking about before is signed by a key, and that key has... There's a search chain, basically, all the way up to a known public Amazon Nitro key. It's kind of like TLS, actually. So we get the attestation document, we... Yeah. Then verify that, "Okay, this is signed by a key, which is signed by a key, which is signed by a key that we have previously known, and it's static for a long period of time.". But at the end of the day, you kind of are trusting Amazon here. You kind of have to. It's this trade-off. It's a very... It's super powerful. If you say, "We're going to assume that this process went off without a hitch," you can do arbitrary stuff. You can really... You can implement whatever you want. You can implement in a black box verification. I mean, it's basically... It's kind of like ZK stuff. You can do inside of an enclave, so long as you trust Amazon. And in our case, probably you have to trust like Intel or whatever kind of chip manufacturer is running it. Qualcomm, so...

Nicholas: Mm-hmm. Which I guess is the same problem. I mean, it's the problem of any private/public key pair. At some point, you have to trust the public key, the person who gave you the public key. You need to be verified in person or something. You need to have some way of deciding and then you're trusting. Yeah, yep. And at the same time, whatever device there, they generated that unless they did it on paper. And even then, the hardware on their iPhone could be compromised also by the supply chain. So I guess you're always kind of trusting someone.

Jack Kearney: Yeah, totally. And I think the way we think about it is just who you trust. and how you trust them really just depends on the context of the application you're building or the security, the sensitivity, basically, of what you're doing. So if you've So, like, you know, I would say something like Bitcoin or Ethereum L1, like, definitely, definitely should not be built on this kind of technology. But, like, it is a fast, very, very secure solution for, you know, things that don't have to meet the, like, security bar of potentially replacing the global economy. So, you know, I think it's, you just have to make trade-offs. In our case, we think this is a really reasonable trade-off to make because of, you know, all the awesome properties we get alongside it.

Nicholas: And I guess the point is that for the customer, the alternative is running something themselves, which is not more secure than this.

Jack Kearney: Yeah, yeah.

Nicholas: There's no alternative where you avoid trust dependency. I guess the big question for this application is, do you trust Amazon's HSM setup? But. Yep. At the end of the day, you're going to have to, even if you roll something yourself, you're trusting your setup. Or if you use a different provider, their setup. I don't see a version where there's no, I mean, even if you run it, like, not in the cloud, you're trusting your local setup. Totally.

Jack Kearney: Or even if you're, even if you're, you know, you've generated a seed phrase and you're running MetaMask on your phone, what are you trusting there? You're trusting Apple. You're trusting MetaMask supply chain. You're trusting, yeah, you're trusting, like, quite a lot of other things in that hole. Kind of situation. You're trusting, I'm not sure, actually, the manufacturer, the secure enclave on the iPhone, but you're probably trusting that in a number of places. Like, there's trust assumptions everywhere in crypto. And I think people like to kind of break it into these really clean lines of, like, decentralized, centralized, like, you know, trustless, like, fully self-custodial versus not. And, like, I just, it's so much, like, more, I think, like, complicated than that kind of. You know, dichotomy would suggest.

Nicholas: I thought about it a little bit, but it seems to me like one of, one existential threat to a blockchain that couldn't be overcome by a fork, like a layer zero. social fork is, you know, like the DAO fork, we had the opportunity because of the time delay before the hack could be finalized, that there was an opportunity to intervene. But if something were wrong with, let's say, ledgers or, I don't know. Very popular algorithms for generating the private keys that are commonly used across Ethereum or Bitcoin. If they were compromised in such a way that you could derive the private keys for specific public addresses, then there would be no opportunity to transition to a new setup via a fork, because how would you ever know who was the owner of a given public address or wallet or, I guess, account in Ethereum? Because anybody could then. Generate the private key back. So you would have no way to prove that they were the one post fork. Did you ever think about that?

Jack Kearney: Yeah, absolutely. I mean, that's, I mean, it's a terrifying thing. And it's, I think, you know, people are like, I think that's, that's probably it's, it's in the case of 7212. It's, it wouldn't be like every single wallet or every single kind of, you know, authentication mechanism in Ethereum. But I would suspect that that's a pretty big piece of the concern is, is like. Okay, this is introducing, you know, another curve. And if this curve is introduced, it has the, and there's some vulnerability that's discovered or whatnot. This is a really substantial flaw in like the overall crypto network.

Nicholas: Particularly around that, that curve, which is, its provenance is doubted by some people. Okay, Bryce, I want to talk to you more about recovery and what the user experience is around all this stuff. But before that, I just wanted to take a second to, you know, there's no sponsor for this episode. But Gitcoin. Gitcoin 19 just finished. And this show had a lot of great support. So I really want to thank everybody who donated, who contributed to the first Gitcoin ever for Web3 Galaxy brand. I really personally appreciate it a lot. It's very meaningful. Just that anybody, anybody wants to contribute and listen to the show. I'm always pleasantly surprised when, when I hear some of the brilliant people who listen to the show. So thank you for listening. Thank you to everybody who contributed. And also very special thanks to BuildGuild and Austin Griffith and the BuildGuild community. And by extension, Ethereum Foundation. The foundation who fund that project, who have been supporting this podcast for several months now, four or five months, something like that. So thank you to BuildGuild. Thank you to the contributors. And thank you to everybody who listens. And of course, thanks to the guests, too. All right, I'm finished. So, Bryce, tell me about the recovery story. What's the idea for user recovery? How does Turnkey help someone who wants to recover their PassKey based wallet?

Bryce Ferguson: This is something we've talked a ton about internally. And there are just a lot of trade-offs in the different directions that you can go. And I think with recovery for something like PassKeys, you need something that is kind of requires no setup. Any average consumer is going to have familiarity with it. And so the first recovery mechanism that we rolled out here is just email based recovery. I think for most low value users, this makes a lot of sense. If they lose their PassKey, they get a one time password sent to their email. And then they can add another PassKey. The nice thing here is we give users the opportunity to evolve out of this over time. So you might have a high value user that says, hey, I've actually set up three different PassKeys. I have my backup YubiKey that's stored in a safe at home. I'm comfortable having one hundred thousand dollars on here. I don't want email recovery on here. I don't want to link my security to my email. And so, again, we kind of get back to this principle of. It's not going to be one size fits all. There's a lot of flexibility here. Our policy engine allows for people to build things like social recovery as well, where you could just have multiple folks who have different PassKeys that are able to together access that private key or add another PassKey that can access that private key to get you back into your account. And so we think that there are just going to be a lot of really novel setups and novel user experiences that evolve out of this over time.

Nicholas: Yeah, definitely. I think the multi-PassKey, multi-factor as a way to get out of the limitations of having some centralized. I'm curious for the email recovery solution, the default that you have so far, at least, is that using the technology that Jack mentioned earlier to like basically trust the authentication? I guess when you bring the secret code in the email to whatever the website that's implementing Turnkey, is it being verified inside of the security? Yeah, it is.

Jack Kearney: It is, definitely. So every form of authentication basically follows that same pattern. It's essentially client-side, like a two-step process. Client-side, you generate a share of the secret that gets transported up along with like a knit recovery sort of process. And then there's a secret that is then also generating the enclave. And then it's basically like Diffie-Hellman. You're like, you get a shared secret then between the two after you do the final leg that's sent to the email address. So Turnkey in the middle basically can't be involved in that process or the intermediate part of our architecture cannot be in the middle of that process. Only the enclave is able to see the secret in full and only the user then is able to also see the secret in full.

Nicholas: So you mentioned that a couple of times, but basically the point is like, even if there's a problem with the API layer of Turnkey, some malicious. Act or something, it wouldn't matter because the secure enclaves are not going to execute code that has not been provisioned by the YubiKeys through that whole process.

Jack Kearney: Yeah, exactly. And that's like our overall threat model actually is. you could fully own every single server in our infrastructure other than a secure enclave, other than these secure enclaves, I guess. And you would not be able to take action on behalf of the user's account. So you couldn't like steal funds or anything like that.

Nicholas: And when there are applications that have to speak to each other, like you mentioned, the EVM part. We're seeing in the signer talking to the policy engine, are they running inside the same like VM in the same secure enclave or can they interact across secure enclaves?

Jack Kearney: So they cannot, they, they are not running the same VM. Okay.

Nicholas: Yep.

Jack Kearney: There's kind of like a broker in the middle there and they'll like each, each of the enclaves other than the core secret is, is totally stateless. So like the broker will be like, I want to do something. I'm going to hand it off to the EVM parser, get the response back. Now I'm going to hand it along to the policy engine, get the response back. And stuff.

Nicholas: I guess we kind of like implied it by our just conversation about pass keys there, but there is no need for a smart account here for a four, three, three, seven style or, or alternative smart contract account, because you have a K one signer in the cloud in a secure enclave. So you can just have a pass key that signs. And I think you mentioned it earlier, but it just feels like you're signing directly with a pass key.

Bryce Ferguson: Yeah. I think that's kind of the magic thing about turnkey is a lot of the promise. Of four, three, three, seven, we've moved forward and a slightly more centralized way. And I think as four, three, three, seven matures and more tooling gets built out around that, we're going to incorporate a lot of that into our experience. We already have partnerships with folks like zero dev that allows people to use those alongside turnkey, but a lot of things like multi-sig and policies you can do out of the box with turnkey.

Nicholas: Awesome. We didn't talk about it yet, but I saw the pricing. It's like 25 cents per signature. Is that not a little high?

Bryce Ferguson: So that, that is just for very low signature amounts. Um, it actually, for enterprise tiers goes as low as 0.001, five dollars. Um, really how we're thinking about this is we're just putting a margin on top of cost of compute as we start to get the scale and want it to be very attractive for pretty much any use case.

Nicholas: Is it expensive to do all this stuff?

Bryce Ferguson: Overall? No, I think we, we've built very, very scalable infrastructure with it.

Nicholas: So I guess once you have more users, the sort of minimum pricing might come down. Yeah.

Bryce Ferguson: I could imagine that happens over time for sure.

Nicholas: I'm trying to think how this fits into the whole, in the L2 space. I've been thinking a lot about how sequencers, paymasters and bundlers in the 4337 space may sort of merge into just all happening at the sequencer. Like you can imagine someone like base just doing paymaster subsidies for applications that they want to run on their chain or something like that. I'm wondering how the, who's ultimately going to pay, is it an app developer or someone like Polychain Labs who's going to pay for the service or someone a level above them? Who do you think?

Bryce Ferguson: Yeah, it's a super good question. And I think there are probably going to be some different use cases as time goes on. You could even imagine end users paying for their own own wallets on turnkey in a far off future. But right now it's really just the developers and application builders who are going to be. Paying for that.

Nicholas: I think we've got to the end of my list of questions, but I'm sure there's other topics. Is there anything that we haven't covered that comes to mind that you think is important people know about?

Bryce Ferguson: I feel like that covered the big pieces there. I think just to underscore one of the coolest things about turnkey is just very, very flexible infrastructure. And so we're pretty excited to see the different directions that people go with this. I think every sales conversation that we have, we have pretty. Unique use cases and setups that people are exploring. And there even are some non cryptic use cases that we're starting to hear about. And so I think it really feels like it's just the beginning with turnkey. And over time, the surface area of what's possible on our platform is just going to explode.

Nicholas: I think there's a lot to think about here in terms of both pass keys feeling like regular Ethereum wallets or Ethereum signers. Both. Directly and also on smart accounts, there's a lot of applications. Also, I think that there's a whole other dimension here. that's not really consumer oriented, but just thinking about managing deployments of massive amounts of private keys. It seems to me like doing it via something like this would be much more secure than doing it on your own machine or in some VPS or something like that. And I can't even imagine what kinds of, I don't know, MEV applications or or cross chain applications you might get up to with that. But very cool.

Bryce Ferguson: Totally. Yeah, it's always shocking to me. Just the the ways that people are storing private keys today and a lot of them are just holding them raw or, you know, managing 50 different ledgers. There's a good story at Coinbase of a fund where their office caught on fire and the CFO had to run back in to get the ledgers out of the safe. Oh, no. But yeah, there's stories like that left and right that people are definitely not taking security in the right direction there.

Nicholas: It's interesting. you mentioned the safe. I talked to someone about safes and I don't know if this is the like a physical. Safe and I don't know if this is the industry practice or what, but this person told me that you're supposed to spend 10% of the value of what you're putting into the safe on the safe itself and that no safe in the world can stop. I mean, think about think about it doesn't make any sense and that at least for sort of home safes. I mean, I don't know about a bank. Maybe it's different story, but that they really can't prevent anyone from getting in for more than four hours. I don't know if that's true.

Jack Kearney: I mean, I think safes overall are just way, way less secure. Than you think they'd be. one of our main security engineers, a guy, Lance Vick has just hammered this point home to me time and time again. And like he sent me like a number of YouTube videos. I'm sure you can find them that show people like going to, you know, Home Depot and getting a couple hundred dollars safe off the shelf and just literally taking like a circular saw and just like sawing through it. Like they're just like, it's just like a joke basically. Yeah.

Nicholas: I was talking to someone who sells safes and this guy told me that. It's only four hours, which made me think, I mean, you can't be further than four hours away from home then, or wherever your safe is stored. If you give them four hours, then it's over. And that's assuming that it can't be physically removed. I guess there's some systems where they sort of embed them in concrete or whatever, make it difficult to move them. But, uh, right. Yeah.

Jack Kearney: I think the thing that safes do give you, which is really critical is evidence of it being tampered with, which, you know, is, is, can be a really meaningful thing. In the crypto context, if you've, you know, split up a private key and kind of, you know, sharded it to a bunch of secrets using some mirrors or something and distributed individual components to a number of safes, you might then be like, oh, wow. Safe in our, our warehouse in, you know, Los Angeles was, was broken in and it's really, it's a one-way operation to like break open a safe probably. So like that might then mean that you accelerate your, your effort to like go retrieve the remaining two. Shares in, you know, warehouses and other parts of the country for key rotation.

Nicholas: Also things like tamper proof, uh, envelopes, um, maybe useful for things like that too.

Jack Kearney: But then you get into, you know, acetone and taking apart the tamper proof evident, uh, envelopes and having it not be obvious, but yeah, this is a whole rabbit hole.

Nicholas: I've seen, I remember I saw one person with a prototype and unfortunately I think the, the company never happened, but he had sort of demonstrated how to. Tamper invisibly with tamper proof envelopes and then had developed his own system that would really be much more secure for, you know, any kind of exacto or resealing evidence. Uh, but, uh, yeah, again, it's like yet another thing where maybe solutions like turnkey kind of can simplify some parts of this and also things like multi-factor, uh, like having a bunch of different devices with pass keys or like the UV key set up that you've got. It's, it's very interesting. Are they, all of this reminds me of, um, the Jameson lop. I always think of his two things, the hardware wallet test. Thing that he's done. That's so great. And also the, uh, Bitcoin physical attacks, uh, GitHub page. Uh, have you seen that?

Jack Kearney: Oh yeah. Yeah. The physical attacks one is terrifying.

Nicholas: I feel like it's very sobering information, uh, for people like people sometimes don't realize as they walk around, I don't know, crypto conferences or flex their Lambos and, and NFTs at the same time. Um, I feel that document is really important. Are there any other resources like that, that you two think about?

Jack Kearney: Um, I mean, the fit, the, like the, the page on Bitcoin attacks, like known Bitcoin attacks is, is huge. I think one that's, that's maybe slightly different here is, um, there's the, the crypto graveyard, I think is a really interesting, uh, website that I, I think is no longer really maintained, but it, it sort of documented like all the different crypto companies that have gone under for massive security issues. Um, and yeah, I mean, that's sort of off topic here, but. There's. A long list of other ways to, you know, shoot yourself in the foot when you're dealing with crypto.

Nicholas: Absolutely. I was thinking about this as people are talking about modifying the EVM for different L2s and especially how it manages state, like it manages storage and memory and stuff. It seems like the number of foot guns, I guess that was one question I didn't really cover when we were talking about the technical stuff, but do you ever feel like this is so complicated that the surface area is, is just dangerous itself?

Jack Kearney: With turnkey?

Nicholas: With turnkey or any kind of, I'm not even turnkey specifically, but just any kind of solution. It attempts to do, you know, to provide something like a cloud, a private key API or something like that, or re or secure enclaves in the cloud, HSMs, et cetera, to, there's so many moving parts, uh, maybe obscurity is one aspect, just, it's so, there's so few people who can understand the system even to break it, but do you worry about like, is simplicity a value in this part of the, of the development world?

Jack Kearney: Oh, absolutely. I mean, like Corma West, like a big part of why it's, um, you know, I think as, as good as it is, is because it's so simple, like we've just stripped out as much as possible from it. It's kind of like, like Bryce and I sometimes have this metaphor to talk about turnkey where it's like, it's an iceberg or like you look at our API surface, it's not actually that complicated. Like there's not that much you can do with it. And it's only when you sort of start peeling apart the layers that you're like, wow, this is really complicated under the hood. Um, or, or. Underneath the surface of the water. And I think, I think that's kind of the balance here is like, it is really, really complicated to do this well. So what you ultimately surface in a product has to be really simple.

Nicholas: It's very interesting. Uh, if people want to check out more about turnkey or I don't know, even maybe participate in some way in the open sourcing of stuff or, or developing applications on top of core MOS, where should they go? What should they learn more about?

Bryce Ferguson: I think a great place to start is just our website and docs. So turnkey.com. Um, if you're interested in collaborating on some of these things as well. You can always reach us, reach out to us at hello@turnkey.com. So very responsive there. Um, but our docs go very, very deep on this and folks can sign up, uh, without talking to anyone to, to play with turnkey itself. Awesome.

Nicholas: And it's, it's live now, right? It's been live for a little while.

Bryce Ferguson: That's exactly right. Yeah. So why the end just takes a couple, a couple of seconds to sign up.

Nicholas: And there's like a 25 signatures per month, three, something like that. So people can even use it without, uh, without paying.

Bryce Ferguson: A hundred percent. Yep.

Nicholas: Very cool. I actually, I forgot to ask how many people are working at turnkey nowadays.

Bryce Ferguson: So right now we are 13 folks, very engineering, heavy team, 10 of the 13 are our engineers.

Nicholas: And is there a central location or remote?

Bryce Ferguson: We're, we're, uh, kind of building out a bit of a hub in New York, but very remote friendly as well.

Nicholas: All right, Bryce, Jack, thank you so much for coming on the show and sharing all this, uh, deep technical information. Uh, I'm still wrapping my mind around it. It's, uh, it's very interesting.

Jack Kearney: Yeah. It was awesome to be here.

Bryce Ferguson: Yeah. Thanks so much for having us. This is great.

Nicholas: Yeah, absolutely. Hope to have you back someday in the future. All right. Thanks everybody for coming to listen. Thanks again, Bryce and Jack. See you next week.

Bryce Ferguson: Happy Friday, everyone.

Nicholas: Hey, thanks for listening to this episode of web three galaxy brain. to keep up with everything web three, follow me on Twitter at Nicholas with four leading ends. You can find links to the topics discussed on today's episode in the show notes, podcast feed links are available at web three galaxy brain.com. web three galaxy brain airs live most Friday afternoons at 5 PM Eastern time, 2200 UTC on Twitter spaces. I look forward to seeing you there. Bye.