Help, I see this error and don't know what it means!
If you have seen a scary ERROR log in your beacon node or validator, there are few things you should do first. Go to our open issues to see if someone has reported the same problem before. Otherwise, note the Prysm version you are running, your operating system, and file a bug report here. You can also ask our community on discord about your error and we will take a look as soon as possible.
My validator is losing money, what's going on?
If your validators are performing poorly and losing money, there are few pieces of information we need to help you out.
Things you can do
- Check if your node or validator client crashed. You can view the logs of the process to see if there were any ERROR logs. If so, please file a bug report or talk to our team on discord. A fatal crash is quite serious and something we will investigate right away. You can try restarting your processes in the meantime to see if you can be fully operational again.
- Check your network connectivity. You can improve this by following our tips here which can help you find better peers, improve attestation effectiveness, and more.
- Check your system resource usage, perhaps your node is using excess CPU and RAM. Depending on your operating system, there are different ways to do this.
Things we can do
If your issue has not yet been resolved, please do the following. Note your Prysm version and operating system, ensure none of the issues are above were the root cause of your problem, then reach out to our team on Discord. We need as much information about error logs or other issues you may be seeing to effectively answer your questions and get you back to validating correctly.
My node suddenly lost peers, what can I do?
Losing peers can be due the following reasons:
- Your network connectivity has problems. You can check how to improve it with some of our tips here.
- Prysm is using a ton of memory or system resources and perhaps you ran out of memory. Ensure you meet the minimum specifications for runnuing Prysm specified in our installation pages for your operating system.
- A bug in our software that can affect your p2p connectivity. It is known that certain versions have issues with peers on operating systems such as Windows, so you could try downgrading to see if your issue is resolved. If this is the case, talk to our team on Discord letting us know you had this issue
I have an issue in the web UI, getting errors, what can I do?
The web UI is still in beta meaning there are bound to be UI bugs or other issues that we need to resolve. If you ran into a bug, please file a bug report here with a thorough description of your problem and screenshots if possible.
My CPU/RAM usage is huge, what’s going on?
Memory management is an ongoing process for our team - we are constantly working on improving the experience of running a node itself. For now, these are the specs we recommend (these values will go down over time):
Recommended Hardware Processor: Intel Core i7–4770 or AMD FX-8310 or better Memory: 16GB RAM Storage: 100GB available space SSD Internet: Broadband connection
Minimum Hardware: Operating System: 64-bit Linux, Mac OS X, Windows Processor: Intel Core i5–760 or AMD FX-8100 or better Memory: 8GB RAM Storage: 20GB available space SSD Internet: Broadband connection
If that still does not help, please file an issue with our team on Github here.
I got slashed! Why? How can I avoid getting slashed?
Slashing is a way for the network to penalize validator actions that can be harmful to the eth2 network. At a high level, a single validator proposing two conflicting blocks or votes or trying to rewrite the history of the chain is considered malicious and such validators will get slashed. Unfortunately, there is no way for the protocol to detect between malicious validators or validators who simply had some faulty configuration that led them to create a slashable offense.
The most common way validators get slashed is by running the same validator key in two separate validator client processes at the same time. This will absolutely get you slashed. Some stakers try to create complicated failover scenarios without realizing the risk this entails, do not do this. If you already got slashed, you will leak funds for a while until you are forcefully exited from the validator registry. Thankfully, slashing penalties in phase 0 are quite small. If you are slashed, you should keep performing your validator duties until you are exited. You will be able to then withdraw your validator balance in future phases of eth2 and will miss out on all the rewards until then.
Our team prepared a blog post on slashing prevention tips you can read to avoid slashings in the future.
How can I check my Prysm version?
Depending on your installation method, there are different ways to verify your prysm version:
- Installation script
How can I upgrade Prysm? Do I just need to close and restart the process?
Upgrading Prysm is done differently depending on your operating system and installation method. We prepared comprehensive instructions here in our docs portal on upgrading and downgrading Prysm.
How can I downgrade Prysm to an older version?
Upgrading Prysm is done differently depending on your operating system and installation method. Please note that downgrading may not be as easy as upgrading as some versions may not be backwards compatible and you will need to perform extra steps. For example, migrating down from v1.x to v1.0.x has breaking changes that require you to also rollback your database.
We prepared comprehensive instructions here in our docs portal on upgrading and downgrading Prysm.
How can I improve my attestation effectiveness?
Attestation effectiveness is a metric that directly affects your validator rewards. In simple terms, an attestation is more valuable the sooner it is put into a block and included in the chain. This interval is called the "inclusion distance" of an attestation. The smaller it is, the more profitable your validator will be. We highly recommend reading Attestant's awesome blog post on the matter here.
Some stakers might notice their effectiveness is perfect, while others might see lower values such as 80%. Improving attestation effectiveness depends on a variety of factors, such as attestation network propagation, your network connectivity, the peers you are connected to. However, your network connectivity is one of the most important factors you can control to improve this metric. We have some tips to improve your connectivity here.
Do I need to keep my deposit keystores after I have imported them into Prysm?
It's always good to keep offline backups of important files such as the original keystores you used to import into Prysm. However, after you import them and are running Prysm successfully, there is no reason to keep the keystore files in that computer as Prysm has already converted the private keys into its own wallet representation. You can delete them.
I am running into issues running geth, how can I get help?
For help with running geth specifically, the go-ethereum discord is the most reliable source of help.
Where can I find the Prysm API documentation?
Prysm uses its own API schema at the moment, called eth/v1alpha1 maintained open source at github.com/prysmaticlabs/ethereumapis. However, there is also a standard eth2 API that all eth2 teams are working on implementing and aligning with. Prysm is still in the process of aligning to this specification and will likely be complete by end of Q1 2021. You can browse the API documentation for this new standard here. Note: Prysm still does not implement this standard but it is priority for our team.
How often should I perform database backups?
The Prysm beacon node and validator allow performing database backups in case your machine gets corrupted and you need to restore it from some checkpoint. You can read our instructions on performing backups here. Briefly, the frequency at which you should perform backups really depends on your personal preference. If you want to perform backups once a day or once every week, ther is no harm nor bigger difference in doing so. Losing your beacon chain database is not a big deal aside from the fact that you will need to sync again from genesis. Losing your validator db can be problematic but if you wait several epochs before starting your validator, ensure your computer's clock is synced, the risk of slashing is low.
Seeing a warning regarding binary signature not being trusted when downloading Prysm, should I be worried?
When downloading the Prysm precompiled binaries such as with prysm.sh or prysm.bat, you might see the following output
Regarding the WARNING above, it means that the signature is correctly signed by the owner, however, since Prysm does not yet provide official linux packages nor windows trusted certs, you will get a warning that it might not have been Preston Van Loon that actually signed the binary, since there is no way to verify the integrity with a trusted authority. If binary integrity and security matters the most to you, building and running open source software from source is the best way to have full confidence the software you are running is a result of code you can inspect yourself.
How can I view the metrics of my node? How can I visualize them?
Your node is running the popular prometheus server for metrics collection which displays its data by default on localhost:8080/metrics. You can visualize these metrics by connecting them to a grafana instance.
Can I run this on a raspberry pi? What are the instructions?
Yes we currently support arm 64-bit architectures such as the raspberry pi 4 and they go out as part of our pre-compiled binary releases here. Our documentation portal has instructions on how to run the entire installation process here. However, we recommend using more powerful hardware in mainnet conditions.
Validator keys and validator deposits
I sent my deposit and so much time has passed but my validator is not active yet
Depositing into eth2 is a multi-step process that can require quite a bit of waiting. As a comprehensive reference, we recommend reading Jim McDee's excellent post on the matter here. If you sent your deposit, we recommend tracking its progress in one of the block explorers such as beaconcha.in or beaconscan. If your deposit is pending and it has still been a while, it is likely because there are a lot of validators in the pending queue ahead of you. There are approximately 4 validators activated every 6.4 minutes, and with thousands in the queue, your wait time can take days or even weeks. You can check how many validators are in the queue in the front page of beaconcha.in.
I made a correct deposit and my validator status in Prysm is still UNKNOWN, what’s going on?
There are a few possibilities. (1) your deposit has not yet been processed by beacon nodes. It takes a while for the beacon node to be able to process logs from the eth1 chain by design. If you have already waited a few hours and no luck, there is a chance that (2) your deposit did not verify (that is, you used some other method of creating the deposit than our recommended, standard way on the eth2 launchpad), or (3) you never actually sent a deposit to the right contract address
How can I move my validator to a different computer without getting slashed?
Prysm will soon implement the slashing protection standard format, meaning that you can export your slashing protection history and import it easily into another machine running any eth2 client, not just Prysm! In the meantime, however, migrating machines can be a little tricky and we prepared the following set of tips to help keep you safe.
- Turn off your beacon node and validator on machine 1, make sure it is not running as a system process. You can check this using the process monitor tools of your OS, or a command line tool such as top or htop and check for anything containing the name “prysm” “validator” or “beacon”
- Note the location of your wallet directory. If you used the default when you started Prysm, you can view its path at the top of the output of
validator accounts list, which varies based on your operating system
- Take that entire directory and move it over to your next machine
- If you modified your validators’ — datadir, also migrate that directory to your next machine
- Wait at least a few epochs, sync your beacon node on your second machine, then start your validator client on your second machine
- Ensure you never run the same keys again on machine 1 or anywhere else
Can I use testnet validators keys on mainnet?
While this is possible, we highly recommend against it. Given there are so many things that can go wrong when generating keys and when sending deposits to eth2, we absolutely recommend starting on a clean slate and not reusing any data you used for your testnet runs. Messing something up is not worth the risk.
Can I send validator deposits using different ETH1 accounts or should it all be from a single metamask?
Yes this is possible however we recommend against it. We recommend you deposit all within the same eth2 launchpad session from a single metamask instance to keep things simple as it is possible a step could get messed up along the way.
Can I add more validators after I am already running one with Prysm?
Adding new validators to your already-running Prysm instance is quite simple! Go through the launchpad process again, generate a new deposit keystore using the eth2 deposit cli, then run the
validator accounts import command again as you used when you added your first validator key. You can then confirm your validator was added by running
validator accounts list. Let us know on discord if you still have issues.
Please note you will need to restart the validator client after importing the new one for the changes to take effect.
How can I get a bunch of test ETH to spin up a lot of validators on the testnet?
Our current faucet only gives out increments of 32.5 Goerli test ETH at any given time and prevents sending out to the same address too often. If you'd like more Goerli ETH, please use our Discord faucet. If you need a larger amount of test ETH, please reach out to us on discord here.
Eth2 specific questions
Why are some validators making a lot more money than others?
If you look at the validator leaderboard, there are some validators at the top that seem to be doing a lot better than others. The reason being that either they (a) got lucky with being assigned to propse more blocks than other validators, or (b) they caught slashable offenses in the network and packed them into their blocks. Slashings are meant to be rare, and Prysm's slasher by default broadcasts slashings it finds to the network so that validators do not selfishly hold on to them. You can actually disable this to selfishly withhold slashings with the
--disable-broadcast-slasings flag in your beacon node, although don't expect to get rich from slashing other validators.
Overall, keep in mind that no one has an extra advantage as a validator compared to others. Block proposal opportunities are random and it does not matter how powerful your hardware is. A lot of the times the validators near the top simply got lucky.
It's been so long and I have not yet proposed a block, when will this happen?
Proposing a block is a factor of random chance depending on the number of active validators currently in the network. As time goes on, however, the probability of you proposing a block indeed goes up. One of our contributors, dv8s, prepared a handy tool to calculate your proposal probability here. Your hardware does not affect your chances of you getting assigned to propose a block, it is random and you cannot gain any advantage over other validators.
Can I get back my ETH? How can I withdraw my validator gains?
Deposits into eth2 are one-way in the beginning. Functionality of withdrawing gains transferring ETH won’t be available until later phases of the project. You can read more about the phases of eth2 here.
Where can I read more about the incentive model in eth2?
This resource from the Ethereum blog is an excellent example.
What is balance and what is effective balance?
Your validator balance is the actual amount of ETH you have from being a validator in eth2. It can go up or down every epoch depending on your participation. Effective balance is a bit different. It is a value that lags behind your actual balance and is used exclusively to determine rewards and penalties for your validator. It has a max value of 32ETH, but your regular balance is uncapped. You can read more about the purpose of effective balance in this excellent post by Attestant here.