Have you ever wondered how fast the dBFT is? In our latest comprehensive study using the enhanced neo-bench tool, we pushed the NeoGo network to its operational limits to identify the minimum possible time that dBFT spends on block acceptance under various load conditions. The results are inspiring: a distributed cluster with 7 consensus nodes can achieve a 16K TPS rate with a median of ~140 milliseconds per block! Interesting, right?

We conducted a study where dBFT was put into extremely time-constrained conditions and stress-tested against the liveness and throughput. Key takeaways of this study include:

• Minimal Reachable Time Per Block Established: Through systematic testing, we pinpointed the lowest median time per block of ~140 ms for clustered environments (reaching ~16K TPS) and ~29 ms for local settings (reaching ~4K TPS). That is effectively the minimum possible block time that the system can function with.
• Minimal Optimal Time Per Block Established: The minimum block time setting showed variable intervals in block production. Therefore, we identified the optimal minimum block times: 300 ms for clustered environments, which supported up to ~17.5K TPS, and 100 ms for local environments, which achieved up to 2.8K TPS. These settings robustly maintained system efficiency without compromising reliability or throughput.
• Robust Under Stress: Even under high loads and extremely low block times, the network maintained significant stability and reliability. It efficiently handled blocks, achieving block times close to the target across both the resource-constrained local setup and the real over-the-internet cluster setup with inevitable and variable communication delays.

Overview

In our continuous effort to push the boundaries of the Neo N3 network, we have embarked on a focused exploration to identify the operational limits under varying block time constraints, utilizing our enhanced benchmarking tool, neo-bench. This detailed investigation is a key part of our strategy to enhance the network’s efficiency and robustness and pinpoint the thresholds where the network’s performance might degrade under pressure. Our primary goal was to investigate the dBFT behaviour and network liveness in time-constrained conditions to squeeze as much from the network as possible with frequently accepted blocks (minimizing acceptance delays for transactions).

The test was conducted using two distinct environments with NeoGo v0.105.1 and neo-bench, which was recently enhanced with new flags (including `msPerBlock`), to analyze the impact of block time variations, specifically:

1. Local Environment Testing:

• Setup: Four consensus nodes and a single RPC node. All are hosted within Docker containers on a single machine (MacBook Air 2020 M1, 8 GB RAM, 512 GB SSD).
• Purpose: This phase was designed to gain a preliminary understanding of the network’s behaviour in a resource-constrained environment with no real network communication latencies between the nodes. We aimed to establish a baseline for how the network handles transaction processing and resource utilization as we manipulated the MillisecondsPerBlock protocol setting in an extremely hardware-constrained environment.
• Methodology: Testing began in a “worker” neo-bench mode continuously pushing as many transactions to the network as the nodes can accept to establish performance baselines with various block times. These tests were followed by “rate” mode testing aimed to provide a specific constant-value load to the network to see how changes in block time affected the network’s ability to handle different load rates.

2. Cluster Environment Testing:

• Setup: An expanded configuration involving eight physical machines (AMD Ryzen 5 3600, 64 GB RAM, 500 GB NVMe SSD) — seven consensus nodes and one RPC node combined with the neo-bench loader instance. Several data centers in two different countries were used.
• Purpose: To emulate a more realistic, decentralized network scenario with network communication latencies. This setup was crucial for evaluating the dBFT liveness and network’s capability under distributed conditions that closely mirror actual usage and potential stress factors in a time-constrained block scenario.
• Methodology: Similar to the local tests, we used worker mode to find the upper limits of performance and rate mode to assess how well the network managed higher transaction rates at minimal block intervals.

The primary goal of these exhaustive tests was to:

• Identify the Breakpoint: Determine the highest load the network can sustain before performance becomes unacceptable with standard block time. This involved pushing the network to its operational limits to find where it starts to fail.
• Minimize Block Time: Subsequently, decrease the MSPerBlock network setting under constant-rate pressure to find the minimum possible time per block that the network can provide.
• Balance Performance and Latency: After identifying the network’s limits, the next step was to find an optimal balance between performance (transaction throughput) and latency (block time). This balance is crucial for ensuring that the network can handle real-world applications efficiently without compromising on speed or reliability.

These experiments are critical for the ongoing development of the metadata handling subsystem for NeoFS networks, but they’re also relevant for other Neo networks.

Benchmark Setup

Configuration Details

• Node Setup: All tests were conducted using NeoGo v0.105.1 nodes, following our focused analysis on these nodes due to their outstanding performance metrics discussed in our latest benchmark post.
• Environment Consistency: For each test run we start from scratch to ensure consistency and accuracy of our results. This involved clearing the environment and resetting the blockchain and network to initial conditions.
• Transaction Generation: Since our primary goal was to investigate network behaviour in a time-constrained environment, we didn’t need sophisticated transactions for tests. Thus, as usual, a batch of simple NEP-17 transfers was generated to simulate network activity.

We systematically reduced the MSPerBlock protocol setting from the default 5000 ms to as low as 25 ms. This range allowed us to explore the network’s responsiveness and stability across a spectrum of block time intervals.

Execution Methodology

The benchmarks were executed in phases to gauge performance across different settings methodically:

• Phase I: Focused on higher MSPerBlock settings to establish baseline performance data on local and external setups under high loader pressure.
• Phase II: Gradually moved to lower MSPerBlock settings to identify the thresholds where TPS values remained unchanged irrespective of the increasing RPS load or the performance started to degrade under constant loader pressure.

Using the benchmark results, we analyzed transaction throughput and system stability and identified the network’s operational limits in terms of minimum time per block and acceptable RPS load. This analysis was crucial in understanding the maximum load the network and dBFT can handle without significant performance degradation in a time-constrained environment.

Results and Analysis

Local Network Setup

Phase I

In the initial phase of local setup testing, we utilized workers mode with the constant number of workers (10) to evaluate the maximum possible TPS at not-so-small MSPerBlock setting values (the range from 1s to 5s).

CNs can easily handle 3684 average TPS at max for 2-seconds blocks and 3079 average TPS at min for 1-second blocks. It should be mentioned that these values are much lower than the 15K TPS got in our previous benchmarks since this time we use different resource-constrained hardware setup. This setup is still sufficient for our needs since our main goal is to get comparable benchmark results.

Note that the TPS spike happens at 2-seconds blocks, i.e. network with a smaller MSPerBlock setting performs slightly better than the 5-second blocks network. This behavior is permissible in the resource-constrained setup, with some variations possible across different benchmarks. At the same time, in the resource-constrained environment, the 1-second block scenario shows lower throughput which may be a consequence of the growing overhead for block acceptance in the hardware resource consumption.

The figure above shows the actual time spent on block acceptance in each of the five experiments. Note that the scatter plot presented above has a logarithmic scale on the MSPerBlock axis since several spikes are rising up to 20 seconds per block.

In general, all series do not contain significant latencies or spikes in the actual block time interval; all series are smooth and close to the desired configured MSPerBlock value. For a 5s target, the network’s median block time is close to 5.44s (~9% higher than configured block time), with less variability, as indicated by tighter 80th (~5.62s, ~12% higher than desired value) and 90th (~5.86s, ~17% higher than desired value). Conversely, at a 1s target, block times are less consistent, with a median of 1.16s (~16% higher than the configured block time) and 1.61s at the 75th percentile (~61% higher than the desired value). However, there are several spikes at the end of every experiment likely denoting ChangeViews performed by consensus nodes due to the lack of memory pool synchronization and processing delays.

Below presented some statistics on the worker mode benchmarks for the local setup:

The highest reachable TPS of 3684 rounded to the higher thousands was taken as the breaking point for the subsequent rate mode testing scenarios. It should be noted that due to the imprecise nature of the load, the target neo-bench RPS level does not directly translate into TPS. Therefore, we are pushing slightly more transactions than the network can comfortably handle. But this value (4K target RPS) is still close to the defined maximum TPS in the workers mode.

Phase II

Starting from the load of 1000 requests per second for 5-second blocks and varying RPS value up to the practically defined breakpoint of 4000 RPS with a step of 1000 RPS, we’ve conducted a set of experiments for network setups with different MSPerBlock settings.

Below is a set of figures for setups with MSPerBlock configured to be 5s, 1s, 600ms, 300ms and 25ms under increasing RPS load. For every benchmarked MSPerBlock target value two plots are presented:

• A low-scaled plot containing measurements for every single block in the test. This plot shows the overall deviation in the block acceptance interval and includes all spikes. This plot also displays the mean and standard deviation of the real block acceptance delays.
• A zoomed plot containing the series with a unified scale with measurements for blocks excluding spikes and strikes, i.e. the most valuable part of the first plot. This plot also includes some statistics (median and 80, 90, and 95 percentile scores) calculated based on the whole set of measurements.

5-seconds blocks

The first analyzed setup is a network with 5-second blocks that showed robustness against 1–3K RPS load and maximum TPS of 3034 under 4K rate load. Under minimal tested load of 1K RPS the network effectively adheres to the configured block interval, demonstrating robust block timing very close to the target. 95% of blocks are accepted within less than 5.03 ms (0.6% slower than the configured value) and it’s clear that no ChangeView happens at all.

However, it’s noticeable that increasing the RPS load to 2K-4K results in larger deviations from the desired block time by the end of each benchmark. Thus, under the 4K RPS target load, only a few blocks are accepted within the same 5.03 ms whereas the mean time of block acceptance has increased up to 5.06 ms (1.2% slower than the configured value) which is still very close to the desired result.

For those who are interested in some more precise numbers, we offer the following table of statistics:

1-second blocks

Further analysis of 1-second blocks shows the best attempt reaching 2544 TPS under the highest load of 4K RPS. The overall trend remains the same: only 1.5% (15 ms) higher block delay for 95% of all blocks compared to the target 1s value under the load of 1K RPS. And increasing deviations from the desired block time for increasing RPS load up to 9% slower than the configured value.

However, from an unscaled plot, it can be noticed that “spikes” appear in the middle of every benchmark, i.e. blocks that were accepted with a delay that’s more than two or even three times higher than the target value. These “spikes” are evidence of change views accepted by consensus nodes and confirmed to be caused by the lack of synchronization between the CNs’ memory pools. The number of “spikes” directly depends on the load level starting from 0.34% of “delayed” blocks under 1K RPS load and ending with ~1% of “delayed” blocks under the load of 4K RPS.

600-ms blocks

Here comes the most intriguing part of our study: we’ve reduced the minimum time per block configuration value to just 600 milliseconds. As you can see from the plots above, the network remains stable and efficiently manages the increasing RPS load levels, demonstrating its robust capability under intensified conditions. Similar to the 1-second and 5-second experiments, the median block acceptance time ranges from 607ms in the best case — 1.1% slower than the desired block time — to 622 ms in the worst case, which is 3.7% slower. These numbers are strongly comparable with delays obtained from more stable experiments with larger block time. Note, that this time the “worst” experiment in terms of both median and percentile block acceptance delay was the one with a 3K target RPS load. This can be explained by the fact that under the high level of load benchmark results become more unpredictable in terms of the number of ChangeViews happening, and this particular experiment was just an “unlucky” one with more than 4% of delayed blocks.

Also notice that the number of “delayed” blocks accepted after the set of dBFT’s view changes increases significantly compared to the previous experiments: from ~1% under 1K RPS load (with resulting 998 average TPS) to ~4% under the worst case of 3K target RPS load (with resulting 1597 TPS). The best TPS value of 2463 is shown under the load of 4K RPS with around 2% of “delayed” blocks.

300-ms blocks

Pushing the boundaries further, we decreased the target MSPerBlock protocol configuration value down to 300 ms. Starting from a 2K RPS load, 95% of blocks are delayed within ~56 milliseconds compared to the target block time (~19% slower than the desired value). The number of blocks accepted from the second attempt varies from an acceptable 0.95% under a 1K RPS load to a maximum of 4.3% under a 3K RPS load level, which is also not a large value.

Also note that block “spikes” caused by view changes organize several well-shaped lines at the level of 800, 1000 and 2000 milliseconds

25-ms blocks

The pattern of 300-ms blocks is being preserved for shorter block intervals below 25-ms blocks: the less MSPerBlock setting is, the higher the deviation from the target block time CNs get. That is, for 100 ms of the target block acceptance interval in the best case, 95% of all blocks are accepted by less than 109 ms which is only 9% higher than the desired value whereas the same percentile score for 25-ms blocks equals 34 ms which is 36% slower than the target value. At the same time, the number of spikes, i.e. ChangeViews accepted, does not grow (4.67% in the worst-case scenario of 100-ms blocks vs 4.2% in the worst-case of 25-ms blocks). The number of RPC errors increases significantly compared to the previous benchmarks.

The charts for the 25 ms MSPerBlock setting are presented in the conclusion of the local benchmarks part since this value was the last one the local network could sustain under the load. It’s noticeable that the median block acceptance time is 16% higher than the desired value, with only 75% of all blocks being accepted within less than 30 ms. However, even with this extremely time-constrained scenario dBFT can sustain the load and show the desired TPS values that are very close to the load rate: 999 TPS under 1K RPS load, 1998 TPS under 2K RPS load, 3000 TPS under 3K RPS load and finally, 3993 TPS under 4K RPS load.

Local Benchmark Summary

Below is a summarized set of 3D scatter plots describing the dependency of block acceptance delays from the network load level. Each 3D plot corresponds to the fixed MSPerBlock protocol configuration setting. Each series in the plot represents a single TPS level reached under the load rate ranging from 1K to 4K requests per second. The plot outline:

• Proximity to the target time per block: The closer each series is to a linear path near the target value across the milliseconds per block axis, the more consistent the performance, indicating stable block acceptance times. A smooth, horizontal line indicates optimal performance, characterized by minimal deviations in block time, representing the ideal case.
• Proximity to the target RPS level: Series at varying heights represent average TPS levels the network is handling. The closer the TPS series aligns with the target load RPS level, the greater the network’s robustness to that load level.

This visual representation helps us quickly grasp the relationship between milliseconds per block and the network load level and assess the network’s efficiency under different load conditions, providing a clear picture of benchmark aggregated results. The key benchmark sum-ups for the 4 consensus nodes local network are:

• The network can sustain the load up to 4K RPS with the minimum tested block acceptance interval of 25 ms. Under these extremely time-constrained conditions, dBFT maintains acceptable performance, producing blocks with a median interval of 30 ms. It keeps the number of ChangeViews below 5% and achieves an average TPS of 3993, precisely meeting the expected level.
• The “ideal” target MSPerBlock value the local network can operate consistently with is 100 ms. This setup produces blocks without significant delay spikes (< 4.7 %) with time per block extremely close to the desired value (< 7% higher than the target delay in the median and ~11% slower than the target delay in the 80th percentile) and TPS value extremely close to the desired one (3111 average TPS at max).

These results are evidence of an outstanding dBFT performance, load robustness and unrelieved algorithm potential that is currently unused (remember, the target MSPerBlock protocol value of N3 Mainnet is 15 seconds). Moreover, local benchmark results gave us hope to get external cluster benchmark results nearly as good as local ones.

Real Network Setup

Once we’ve got these inspiring results from the local network setup, our next goal is to test how well dBFT performs in a real distributed network scenario with a higher number of consensus nodes. Compared to the extremely resource-constrained local network scenario, consensus nodes from the external cluster are not restricted in hardware resources and, especially, in RAM. On the other hand, this cluster has some realistic network communication delays, so one of our goals is to investigate the impact of these real-life communication delays on the block acceptance interval.

We’ve measured the ping response time between the nodes of our cluster, and it turns out that some of them are closer to each other than others. Cluster machines can be categorized into three groups based on ping response times: the first group, being the most remote, has an average delay of ~25.2 ms; the second group has ~0.435 ms; and the third group, the closest machines, has ~0.022 ms delay. Given this fact and also the fact that consensus nodes need to pass through 3 dBFT rounds to accept a block and send at least two additional protocol messages to retrieve unknown transactions (CMDGetData, CMDData), we can evaluate the approximate communication delays between the nodes.

Phase I

Like the local benchmarks, the external cluster benchmarks begin with worker mode to establish the maximum achievable TPS for the cluster in a typical, time-unconstrained environment. We utilized a range of workers, from 10 to 300, and determined the best average achievable TPS: 9.291 for standard 5-second blocks, 14.747 for 3-second blocks, and 18.319 for 1-second blocks.

According to the figure above TPS increases with lower MSPerBlock setting since the consensus nodes can drain the partially-filled memory pool more often than with higher block time (mempool capacity remains the same within all setups). It may seem that TPS increases limitless with lower block time, but that’s not exactly true (you’ll see it in the rate benchmarks below). There’s a point starting from which operational and communication delays affect the network performance so that TPS can’t grow further.

Also, note that this graph is more smooth than the same graph for the local network setup which is a direct consequence of more predictable (and reproducible) network delays and a sufficient amount of hardware resources. Referring back to our previous article where the network with the same standard (50K) mempool configuration reached 15K TPS in the 4-nodes scenario, turns out that these synthetic results are not so far from the reality with the physical network communication delays.

The actual block acceptance delays are not so close to the target MSPerBlock value as for local setup, which is completely understandable due to additional communication delays. The median acceptance time grows from 7.6% higher than the target value for 5-second blocks up to 23% higher for 1-second blocks. Also note that the nodes do not fallback to change views in 2s-5s setups, unlike the 1-second blocks setup where spikes happen at the end of the benchmark. This behaviour is caused by the lack of synchronization between the nodes causing peer disconnections and, consequently, view changes.

Based on the average TPS values we’ve defined 20K RPS as the performance ceiling for further experiments with MSPerBlock configuration in the rate mode.

Phase II

We’ve started from the experimental well-carried load of 4K requests per second and conducted experiments for the range of RPS with a step of 4000 RPS up to the breakpoint of 20K RPS defined in the previous step. Below is a set of figures for setups with MSPerBlock configured to be 5s, 1s, 600ms, 300ms, 100ms and 60ms under increasing RPS load. Like for local setup, for every target MSPerBlock value two plots are presented:

• A low-scaled plot containing measurements for every single block in the benchmark. This plot shows the overall deviation in block acceptance interval and includes all the spikes. This plot also displays the mean and standard deviation of the block acceptance delays.
• A zoomed plot containing the set of charts with a unified scale for the most valuable part of the first plot, i.e. measurements for blocks excluding spikes and strikes. This plot also includes some statistics (median and 80, 90, and 95 percentile scores) calculated based on the whole set of measurements.

5-seconds blocks

Starting with the highest target MSPerBlock setting of 5 seconds, the network displays a relatively stable deviation in block acceptance delay from the target value. The actual median value under the load level of 1K RPS is just 1.9% higher than the desired value whereas 95% of all blocks are accepted for less than 5194ms which is only 3.9% higher than the target value. Under the larger loads, dBFT accepts blocks with ~5.39s median delay which is 7.7% higher than the target value whereas only 75% of all blocks are accepted by less than 5.4s (with a delay of 8% higher than expected). Although these numbers are relatively good for the real network, note how they differ from the local setup results. Due to the network communication delays, we got ~90–400ms median delays which is a multiple of the ping interval presented above, which is an order higher than 20–60ms median delays for the local setup.

When it comes to TPS, for the worst load case under the 20K target RPS load, the network can successfully sustain only ~9.3K RPS and produces 9282 TPS on average which is the maximum value for this setup and selected MSPerBlock configuration. It should be noted that the theoretical maximum of this setup is 10K TPS since we use a standard mempool size of 50K with 5s blocks. Thus, the resulting values are pretty close to the theoretical maximum. Also note that no change of view happens, which is a desirable behaviour.

1-second blocks

Decreasing the target MSPerBlock value down to 1s gives an interesting side-effect: it’s visible that block acceptance delays are split between two levels that are ~40–50ms far from each other. These levels are closely related to the network delays and correspond to the consensus nodes grouped by their ping response time. For example, with a 1K RPS load, the block acceptance time averages 1090 ms if the dBFT round’s primary node belongs to the ‘closest’ group of consensus nodes (CNs) with a ping time of 0.02–0.4 ms. However, it extends to about 1130 ms if the primary node is in the ‘most remote’ group with a ping time of approximately 26ms.

1s-blocks setup shows robust proximity to the desired block time with a median of 1094 ms (9.4% higher than the desired value) and 95% of blocks falling within the range of 1164 ms delay (16% higher than the desired value) under 1K target RPS load. The median block time exceeds the desired value by 11.3%, 13.6%, 15.9%, and 19.5% for network loads of 8K, 12K, 16K, and 20K RPS, respectively.

The highest TPS achieved is approximately 15.4K under the 16K target load level, significantly below the maximum expected value. Additionally, the percentage of blocks delayed more than twice the MSPerBlock value increases from 2.6% at a 16K RPS load to 8.6% at a 20K RPS load.

Also, under the high load (16K and 20K RPS) it’s noticeable that at the end of the benchmark, there are a couple of spikes exceeding the target MSPerBlock value more than six times. The reason for these spikes is ChangeViews caused by the peers’ disconnection due to the lack of synchronization (too stale or too new consensus messages are considered as the reason for peer disconnection). This can be improved in newer NeoGo versions.

600-ms blocks

And now, we have reached the most intriguing part of our study: we’ve reduced the target MSPerBlock value to just 600ms. The overall trend remains consistent: the network handles increased pressure well, though block acceptance delays fluctuate more under higher loads. The maximum average TPS has increased up to 16764 compared to the previous benchmark. Also, note that the difference between the highest and lowest median block acceptance time within different load levels becomes almost insignificant, there’s a clear tendency for block time delay unification. Thus, under the load of 4K-12K, the median block time slightly grows from 688ms to 713ms exceeding the desired value of 14.5%-18.8%. However, under the high load of 20K RPS, the standard deviation value grows up to 681, and it’s clear that there are more fluctuations in the actual block time values. In the worst scenario, only 80% of all blocks are accepted within less than 760ms (27% slower than desired).

The notion of two block acceptance delay levels becomes clearer with the same ~40–50ms distance between the levels which is another confirmation of the theory about block time dependency from the network communication delays.

The number of ChangeViews, which indicates network adjustments during transaction processing, increases from 4.3% at a load of 16K RPS to 7.4% at a load of 20K RPS. These values are still acceptable for the real network to continue further experiments.

300-ms blocks

Going far beyond the limit, we’ve further decreased the target MSPerBlock protocol configuration value down to 300ms. The median block acceptance delay varies from 380ms (~26.7% higher than desired) at min to 419ms (39.7% higher than desired) at max. Starting from a 4K RPS load, 95% of blocks are delayed within ~121 seconds from the target value (~40.3% slower than the desired value). The number of blocks accepted from the second attempt varies from 8.95% under 16K RPS load to the maximum of 17.6% under 20K RPS load level, which is quite a large value. The maximum average TPS value reached is 17516 TPS under the load of 20K RPS.

This setup is deemed the last practical one, as further reducing the MSPerBlock value leads to block acceptance delays that approach or exceed the configured block time itself, rendering it impractical. However, we’ve included two more benchmark results with lower MSPerBlock values in this article since there are a couple of noticeable patterns found.

100-ms blocks

The 100ms-block setup is included in this post to demonstrate that dBFT can effectively handle small block intervals in a real network environment, adhering to the configured settings even under extremely time-restricted conditions that are comparable to network communication delays. The median block time under 4K-16K load (181–189ms) is still not even twice larger than the target value which means that almost all blocks were accepted without view changing, i.e. from the first attempt of CNs to agree. These values are not inspiring from the practical point of view (no one needs more than 80% delay from the target value for the real system), but at the same time, this experiment is an outstanding challenge for the dBFT algorithm that no one has ever tried before.

This experiment also shows the third level of block acceptance delay under a 4K RPS load. This level indicates the blocks accepted after ChangeView happened.

60-ms blocks

The last experiment in this series of benchmarks is aimed to prove that even dBFT has its performance limit. Once the MSPerBlock value approaches the network communication delays order, dBFT is not capable of accepting blocks without changing its view and increasing its timer for every block. Even though with targeted 60ms per block consensus nodes delays blocks (minimum block acceptance time is around 140ms which is twice as much as the target value), dBFT is still capable of block processing. It still shows robustness against the load pressure and reaches TPS values close to the target RPS load level. The maximum average TPS value reached in this setup was 15974 which is comparable with the previous results. Additionally, although each block requires at least one view change to be accepted, the overall block acceptance delay remains relatively consistent and adheres strictly to rules dictated by network delays — particularly evident at lower RPS loads.

Real Network Benchmark Summary

Below is a summarized set of 3D scatter plots describing the dependency of time per block from the network load level. Each 3D plot corresponds to the fixed MSPerBlock protocol configuration setting. Each surface in the plot represents a single resulting average TPS value reached under the load rate ranging from 4K to 20K requests per second with a step of 4K RPS. The plot outline:

• Proximity to the target time per block: The more linear close to the target value each series is across milliseconds per block axis, the more consistent the performance, indicating stable block acceptance time. A smooth, horizontal line corresponds to the optimal performance without significant block time deviations, which is an ideal case.
• Proximity to the target RPS level: Series at varying heights represent actual TPS levels the network is reaching. The closer the TPS series is to the target load RPS level, the more robust the network is to the load level.

The most significant results that should be emphasized for the 7 consensus nodes of the external network setup are:

• The network can carry the load of 20K target RPS showing the best average TPS of 17516. This value is noticeable since it’s shown on the real network and comparable with the synthetic results obtained during our previous benchmarks.
• The “ideal” target MSPerBlock value the network can operate consistently without a high number of ChangeViews is 300 ms. This setup produces blocks with an not so high number of huge delay spikes (0–18% depending on the load rate). The resulting time per block is not so close to the desired value, but still acceptable (~26.7% slower than the target delay in the median for the best case and ~39.7% slower than the target delay in the median under the full load). This setup is the one that shows the best average TPS within the whole set of benchmarks.
• Under the extremely time-constrained condition of a 60ms minimum target block interval, dBFT demonstrates robust performance. Despite accepting ChangeViews for all blocks, the network maintains a median block acceptance interval of approximately 139ms, indicating stable and acceptable behavior. The maximum average TPS achieved in this setup is 15974, approaching the optimal performance limits.

These results confirm that the dBFT protocol is extremely robust against the time-constrained network configuration in real distributed conditions with significant network communication delays.

Conclusion

Through rigorous benchmark testing using the enhanced neo-bench tool, the NeoGo network demonstrated remarkable performance and stability, revealing critical insights into its capabilities and limits under extremely time-constrained conditions. We established that the real distributed network from 7 consensus nodes could sustain a throughput of up to ~17500 Transactions Per Second (TPS) in a clustered environment effectively outperforming the synthetic results from our previous post.

Key findings from the study underscore the network’s efficiency in handling transactions, even with significantly reduced MSPerBlock settings, demonstrating robustness under high load conditions and extremely low block time scenarios. Notably, the network maintained transaction processing efficiency with a minimum median block time of approximately 140 milliseconds in clustered settings and even faster in local configurations, demonstrating the system’s adaptability and throughput potential.

Finally, the most significant discovery from these experiments is that as the load on the network increases, there is a smooth and predictable degradation in performance. Despite the growing block acceptance delays and more frequent ChangeViews under high loads, the network still functions without sudden failures or unexpected spikes in block acceptance delays. This demonstrates the network’s high reliability and resilience.

The network can adequately respond to the load — it may experience some delays, but resulting performance parameters still fall within acceptable limits and do not lead to critical failures. The network is predictable and that’s exactly what we want from it.

This is an announcement of NeoFS HTTP gateway deprecation. It will no longer be maintained, please migrate your applications to the new updated NeoFS REST gateway. We’ve simplified this transition for you by implementing the same API as was previously provided by the HTTP gateway (see migration guide).

Previously we have also provided public test instances of HTTP gateway for testnet (http.t5.fs.neo.org) and mainnet (http.fs.neo.org). While they were always provided as test-only with no availability guarantees we also know that many community members use them from time to time for various purposes. To make this migration easier for everyone we will keep these domains functioning for at least half a year (that is till October 2024), however the implementation behind them will be transparently switched to the REST gateway:

• http.t5.fs.neo.org will be switched at 12:00 UTC March 20, 2024
• http.fs.neo.org will be switched at 12:00 UTC March 20, 2024 (if there are no problems found after the http.t5.fs.neo.org switch)

We don’t expect this process to affect any applications, however if you notice any problem please contact us via GitHub or via ops@nspcc.io.

Long-awaited 3.6.0 Neo C# node release has introduced interoperability functionality for arithmetic operations with BLS12–381 elliptic curve points that is also supported by the fully compatible 0.102.0 NeoGo node version. In other words, starting from the latest release (already available on mainnet!) all Neo dApp developers have an ability to make use of the BLS12–381 compatible zero-knowledge proof systems natively on the Neo blockchain. In this article we’ll show how to build an end-to-end dApp that generates and verifies Groth16 proofs in the Neo ecosystem. It is written in Go and uses NeoGo, but the mechanisms used are language-agnostic (as all things in Neo are).

Disclaimer: this article leaves the caveats of the Groth16 algorithm out of the scope, concentrating instead on Neo implementation specifics. For those who are interested in advanced math, ZK-SNARKs and other ZK-related topics we recommend diving into the list of links prepared by matter-labs.

Content

Zero-knowledge protocol is a method by which one party (the prover) can prove to another party (the verifier) that a given statement is true while avoiding disclosure of any other information beyond the fact that statement is true to the verifier.

Given the definition above, in this article we’ll show you how to create and verify zero knowledge proofs over the BLS12–381 elliptic curve with the Groth-16 proving scheme on the Neo blockchain.

The article contains:

1. Circuit implementation of an example cubic equation composed with the help of Consensys gnark high-level API for circuit design. Translation of the composed circuit into a set of mathematical constraints. A set of unit tests demonstrating how to check the circuit correctness.
2. Golang Verifier smart contract generation for the compiled constraint system and verifying key based on the NeoGo zkpbinding API. Verifier smart contract compilation and deployment to the testing environment using NeoGo neotest package.
3. Proof of knowledge generation for a set of given public inputs and corresponding secret variables with the help of Consensys gnark/backend package. Converting the resulting proof to the compatible format acceptable by the Verifier smart contract with the help of NeoGo zkpbinding API.
4. On-chain proof of knowledge verification via deployed Verifier contract invocation with a set of public input witnesses and generated proof.
5. Tips and tricks for production implementation.

Circuit design

As mentioned above, from a user perspective, a circuit can be described as an algorithm for which you want to prove and verify execution. This circuit can be written in any programming language, but the most common are Rust and Go. However, under the hood a circuit represents a constraint system which is effectively a set of algebraic constraints. For an example algorithm that needs to be proved and verified, we’ll use a simple cubic equation: y = x³ + x + 5. In this case the prover wants to prove that he knows the solution of the mentioned cubic equation without revealing the solution to the verifier.

To define the constraint system we’ll use gnark, an open-source fast ZK-SNARK Go library managed by Consensys: https://github.com/Consensys/gnark. It allows to compile circuits from a high-level Golang API declaration into a constraint system for a wide range of elliptic curves and proving systems: https://docs.gnark.consensys.net/HowTo/write/circuit_structure. Please, refer to the gnark basic concepts for more details on circuit definition and structure and to the gnark documentation for the implementation notes.

The circuit corresponding to the provided cubic equation has the following structure:

package cubic_circuit


import (
   "github.com/consensys/gnark-crypto/ecc"
   "github.com/consensys/gnark/backend/groth16"
   "github.com/consensys/gnark/frontend"
   "github.com/consensys/gnark/frontend/cs/r1cs"
)


// CubicCircuit defines a simple circuit x**3 + x + 5 == y
// that checks that the prover knows the solution of the provided equation.
// The circuit must declare its public and secret inputs as frontend.Variable.
// At compile time, frontend.Compile(...) recursively parses the struct fields
// that contains frontend.Variable to build the frontend.ConstraintSystem.
// By default, a frontend.Variable has the gnark:",secret" visibility.
type CubicCircuit struct {
   // Struct tags on a variable is optional.
   // Default uses variable name and secret visibility.
   X frontend.Variable `gnark:"x,secret"` // Secret input.
   Y frontend.Variable `gnark:"y,public"` // Public input.
}


// A gnark circuit must implement the frontend.Circuit interface
// (https://docs.gnark.consensys.net/HowTo/write/circuit_structure).
var _ = frontend.Circuit(&CubicCircuit{})


// Define declares the circuit constraints
// x**3 + x + 5 == y.
func (circuit *CubicCircuit) Define(api frontend.API) error {
   x3 := api.Mul(circuit.X, circuit.X, circuit.X)
  
   api.AssertIsEqual(circuit.Y, api.Add(x3, circuit.X, 5))
   return nil
}

The circuit itself is represented by the CubicCircuit Go structure that implements frontend.Circuit interface with a single Define(frontend.API) error method, describing the set of circuit constraints. The circuit declares its public (i.e. available both to prover and verifier) and secret (i.e. available only to the prover) inputs as frontend.Variables.

The declared circuit should then be compiled into a rank-1 constraint system (R1CS) over the BLS12–381 elliptic curve scalar field using the following gnark library API. Note that here and below error handling code is omitted.

var circuit    CubicCircuit

// Compile our circuit into a R1CS (a constraint system).
ccs, err := frontend.Compile(ecc.BLS12_381.ScalarField(), r1cs.NewBuilder, &circuit)

After that the resulting constraint.ConstraintSystem can be used to generate proving and verifying keys and build proofs.

For simple constraint systems like our cubic circuit, the implementation is rather trivial and doesn’t require any tests. However, for more complicated circuits it would be nice to ensure that everything works as expected and even perform some testing proof verification. It can easily be done using the gnark testing engine. The following code snippet demonstrates how to create Go unit tests for the circuit and verify it against a set of various elliptic curves and proving algorithms based on certain input data. Please, refer to the NeoGo cubic circuit tests for more circuit unit test examples.

// TestCubicCircuit_Verification performs the circuit correctness testing over a
// set of all supported curves and backends and over a specified curve with a
// set of exact input and output values.
func TestCubicCircuit_Verification(t *testing.T) {
   // Assert object wrapping testing.T.
   assert := test.NewAssert(t)


   // Declare the circuit.
   var cubicCircuit CubicCircuit


   // The default behavior of the assert helper is to test the circuit across
   // all supported curves and backends, ensure correct serialization, and
   // cross-test the constraint system solver against a big.Int test execution
   // engine.
   assert.ProverFailed(&cubicCircuit, &CubicCircuit{
      X: 3, // Wrong value.
      Y: 5,
   })


   // If needed, we can directly specify the desired curves or backends.
   assert.ProverSucceeded(&cubicCircuit, &CubicCircuit{
      X: 3, // Good value.
      Y: 35,
   }, test.WithCurves(ecc.BLS12_381))
}

Now, when we have our circuit properly tested and constraint system ready to be used, let’s move on to the proof verification part.

Verifier smart contract implementation

Once the constraint system is compiled, we can build a proving and verifying key pair associated with the circuit. Proving key will be used by the prover to generate input-specific proofs. Verifying key will be used by the verifier to generate a circuit-specific Neo Verifier smart contract and verify proofs of knowledge for the given circuit via the Verifier contract invocation afterwards. Here’s the example of dummy proving and verifying key pair generation via gnark API:

// One time setup (groth16 zk-SNARK), circuit-specific prover and verifier keys generation.
pk, vk, err := groth16.Setup(ccs)

Verifier smart contract represents a simple Neo contract written in Go language that can be compiled by the NeoGo compiler and deployed to the Neo network in a standard way. NeoGo offers a zkpbinding utility package that can be used to generate the circuit-specific Verifier Go smart contract and all the associated files (contract manifest, go.sum and go.mod files) needed for compilation and deployment. Verifier contract generated via the zkpbinding utility contains a single VerifyProof(a []byte, b []byte, c []byte, publicInput [][]byte) bool method that accepts proof of knowledge for the given circuit represented as three serialized compressed BLS12–381 points and the public input information represented as a list of serialized 32-bytes field elements in the LE form. VerifyProof function implements Groth-16 verification algorithm described in the neo-project/neo#2647 issue without any changes. In fact, VerifyProof performs a set of arithmetical operations over provided BLS12–381 points and the public input scalar field elements to verify the given proof. VerifyProof function returns whether the provided proof is valid for the given public input.

The following testing code snippet demonstrates how to generate Verification contract using the NeoGo zkpbinding utility package:

// Create contract file.
tmpDir := t.TempDir()
srcPath := filepath.Join(tmpDir, "verify.go")
f, err := os.Create(srcPath)


// Create contract configuration file.
cfgPath := filepath.Join(tmpDir, "verify.yml")
fCfg, err := os.Create(cfgPath)


// Create contract go.mod and go.sum files.
fMod, err := os.Create(filepath.Join(tmpDir, "go.mod"))
fSum, err := os.Create(filepath.Join(tmpDir, "go.sum"))


// Generate Verifier contract itself.
err = zkpbinding.GenerateVerifier(zkpbinding.Config{
   VerifyingKey: vk,
   Output:       f,
   CfgOutput:    fCfg,
   GomodOutput:  fMod,
   GosumOutput:  fSum,
})

Note, that the usage of NeoGo zkpbinding utility package is not the only way to compose the Verifier contract. You can write it manually yourself in any other programming language supported by the Neo platform (C#, Go, Python, Java, etc.), compile, deploy and invoke it afterward. The structure of Verifier contract is not constrained by any NEP standard and doesn’t require any specific manifest. However, the verification algorithm itself is expected to remain the same from contract to contract, effectively it’s a Groth-16 implementation of proof of knowledge verification process over BLS12–381 elliptic curve described in the neo-project/neo#2647 issue. Since all the Verifier contract implementations share the same batch of code, NeoGo provides a nice tool for the contract template generation. The resulting contract can be compiled and deployed to the Neo network as is without any additional changes (if applicable) or can be used as a basis for more complicated verification contract construction.

Verification contract can be compiled and deployed to the Neo chain in a standard way, e.g. via NeoGo CLI (see the workshop for details on how to deploy and invoke the contract via NeoGo CLI) or via NeoNova NeoGo RPC client (see the workshop code snippet for details on how to deploy a contract with Actor from NeoGo). At the same time, it would be nice to test the auto-generated Verifier contract. It can be done with a little help from the NeoGo neotest framework. The following testing code snippet demonstrates how to create testing environment (similar to a private network) and deploy Verifier contract into it:

// Create testing chain and executor.
bc, committee := chain.NewSingle(t)
e := neotest.NewExecutor(t, bc, committee, committee)


// Compile verification contract and deploy the contract onto the chain.
c := neotest.CompileFile(t, e.Validator.ScriptHash(), srcPath, cfgPath)
e.DeployContract(t, c, nil)

Once the contract is deployed, the verifier can invoke it and verify a set of proofs of knowledge submitted by the prover.

Proof of knowledge generation

Once the prover has a compiled constraint system, it’s possible to generate proofs of knowledge for the corresponding circuit. A set of proofs can be generated for a set of public and private input pairs, but in the example below we’ll demonstrate a single proof generation. For the cubic circuit described above, an example of a valid solution would be the {X: 3, Y: 35} pair where Y value is a public input data known to everyone and X value is the private solution known to the prover only. The prover uses the provided pair and compiled constraint system to generate the proof via gnark API (note, that corresponding backend and a proper scalar field must be used):

// Valid solution.
var assignment = CubicCircuit{X: 3, Y: 35}


// Intermediate step: witness definition.
witness, err := frontend.NewWitness(&assignment, ecc.BLS12_381.ScalarField())


// Proof creation (groth16).
proof, err := groth16.Prove(ccs, pk, witness)

After that, generated proof can be used as an input to the VerifyProof method of the Verification smart contract. However, arguments of VerifyProof must be compatible with NeoVM and the resulting proof structure can’t be passed directly to the contract, it needs a bit of additional conversion. Based on the VerifyProof signature, things we’d like to pass as function arguments are three BLS12–381 serialized points in compressed form and a slice of public witnesses each represented as a serialized 32-bytes field element in the LE form. To convert proof to the suitable form NeoGo offers the following helper:

// Get the public part of input data.
publicWitness, err := witness.Public()


// Retrieve arguments for the Verifier smart contract call.
args, err := zkpbinding.GetVerifyProofArgs(proof, publicWitness)

The resulting args can be directly used to invoke the VerifyProof method of Verifier contract. Thus, nothing prevents the verifier from checking whether the provided proof of knowledge is correct.

Proof of knowledge verification

Given all the previous steps properly completed, by the moment of verification the verifier should have the BLS12–381 circuit-specific Verifier contract deployed to the Neo chain and a proof received from the prover in a suitable format. The verification itself comes down to a simple VerifyProof method call with provided arguments. It can either be a test invocation or a normal invocation via transaction made via any known tool capable of performing contract invocations (RPC call via CURL, NeoGo CLI contract testinvokefunction command, program call via NeoGo RPC client/Invoker/Actor or via NeoC# RPC client or via any other RPC client). Here’s the example of such Verifier contract call made in previously created testing environment using neotest package:

// Verify proof via verification contract call.
validatorInvoker := e.ValidatorInvoker(c.Hash)
validatorInvoker.Invoke(t, true, "verifyProof", args.A, args.B, args.C, args.PublicWitnesses)

As you can see, the expected true result is left on stack after the invocation which means that the provided proof for the given public input data is valid.

Please, refer to the NeoGo cubic circuit example for the full circuit code and end-to-end tests: example.

Notes and caveats for production usage

The Trusted Setup Ceremony and CRS practical usage

The example above was implemented in a testing environment and aimed to demonstrate the end-to-end pipeline of zero knowledge proof generation and verification on the Neo chain. Generation of the proving and verifying keys in the example above requires some randomness to be precomputed (it is done inside the groth16.Setup method call and known as a Common Reference String or a Structured Reference String). If the process or machine leaks this randomness, an attacker could break the ZKP protocol. Thus, for production usage we strictly do NOT recommend this approach. Instead, Groth16 ZK-SNARK circuits usually require an MPC (Multi-Party Computation) Trusted Setup ceremony to generate the parameters that can kick off ZK-SNARKs-based systems.

Any Trusted Setup is organized in two subsequent steps: Phase 1 and Phase 2. The Phase 1 which is also known as the “Powers of Tau” is universally reusable in any point of contribution as an input for any ZK-SNARK. Phase 2, in contrast, must be performed for each individual circuit and requires the response from the Phase 1 as an input. Neo project core developers have decided not to take their own Powers of Tau ceremony; usage of some external trusted source of randomness is suggested to set up the second phase, for example, the response generated by Filecoin’s Powers of Tau ceremony can be used (see the attestations for it). Note that both phases are curve-specific which means that in the case of Neo ecosystem only BLS12–381-specific results are suitable (at least, at the moment of writing). Here’s the set of steps that every dApp developer team should follow for proper CRS setup and reliable proving/verifying keys generation:

1. Take the Phase 1 output of some external trusted Powers of Tau ceremony over BLS12–381 elliptic curve that is suitable for your circuits, i.e. contains computations of as many powers of tau as needed for the most heavy of your circuits (the circuit with the largest number of constraints). For example, we can choose from the response files generated by the Filecoin’s Powers of Tau ceremony (IPFS link, attestations and response files link) or by any other good PoT ceremony that you trust. Please note that this ceremony MUST be taken over the BLS12–381 elliptic curve for Neo interoperable functionality compatibility.
   An alternative to the way mentioned above would be to setup your own Powers of Tau ceremony with required parameters as described in the guide.
2. Once the Phase 1 parameters are chosen, you can move on to the Phase 2 of the trusted setup which is circuit-specific. Contribute some randomness to the response file from the previous step following the test example and documentation (if working with Go stack) or using any other Phase 2 MPC implementation that is suitable for your team, organisation and stack, e.g. Filecoin’s Phase 2 implementation.
3. Generate proving and verifying keys based on the parameters obtained from the previous step. If using Go stack, then take a look at the Groth-16 demonstration example and, in particular, at the production CRS setup in test.

For more details on CRS generation for Neo dApps in production environment see the discussion in the neo-project/neo#2647 issue. Note, that currently NeoGo doesn’t have a fully-qualified CLI utility for organizing the MPC contribution ceremony over BLS12–381 curve from scratch (like it is done for BN254 elliptic curve in https://github.com/bnb-chain/zkbnb-setup), but it’s possible to port the mentioned solution to BLS12–381 curve. Thus don’t hesitate to contact us if you need this feature (issues and patches to https://github.com/nspcc-dev/neo-go are welcome).

Circuit development toolkit

You can use any other tools to design circuits and generate proofs (circom, arkworks, etc.), gnark and NeoGo usage isn’t required for this. However, assuming that NeoSPCC provides the Neo development toolkit for writing/deploying/invoking smart contracts in Go programming language, this article demonstrates the easiest way of integrating the existing Groth16 Golang development tools with the NeoGo toolkit.

More complicated circuits and constraints

The example above has almost no practical value except that it demonstrates how to use the set of tools provided by NeoGo for ZKP dApps integration with the Neo platform. Production environments are likely to demand more complicated scenarios to prove and more complicated circuits to implement, e.g. ZK-friendly hashing algorithms, Merkle proofs verification, EdDSA signature verification, ZK-rollups assistance, etc. All of this (and not only) can be implemented via the powerful gnark circuit design API. For more elaborate circuit examples we highly recommend reading the set of Write circuits documentation articles and carefully reviewing the set of gnark circuit examples.

However, note that while circuits are programmable, they can’t (practically and efficiently) prove any algorithm. The way constraints are represented make some things more natural (“snark-friendly”) to do than others. Please, review the circuit design considerations.

Conclusion

This article is aimed to be the first step for those developers who start their journey into the ZKP universe of the Neo blockchain. Hopefully, this article gives enough information to build your own ZK application for the Neo platform. Please, note that the described NeoGo Groth-16 functionality is relatively new to the project and may be rough on the edges. Subsequent extension and polish of NeoGo ZK functionality is also possible. Thus, don’t hesitate to contact us on Discord or file an issue, any related questions, comments, suggestions and feedback are welcome. Build, test and run your applications with the NeoGo toolkit, happy proving with ZK-SNARKs!

References

• Full NeoGo cubic circuit example: https://github.com/nspcc-dev/neo-go/blob/91c928e8d35164055e5b2e8efbc898440cc2b486/examples/zkp/cubic_circuit/README.md
• NeoGo zkpbinding package documentation: https://pkg.go.dev/github.com/nspcc-dev/neo-go/pkg/smartcontract/zkpbinding
• A set of gnark circuits examples written in Go: https://github.com/Consensys/gnark/tree/master/examples
• Neo ZK-SNARKs research and implementation of arithmetic operations over BLS12–381 elliptic curve points: https://github.com/neo-project/neo/issues/2647
• Groth16 specification and explanation: https://www.zeroknowledgeblog.com/index.php/groth16
• Neo MPC Phase 2 implementation and circuit development toolkit based on the ebfull’s library: https://github.com/doubiliu/phase2/tree/dev

Modern apps and dApps are mostly web oriented. Web applications provide the same user experience for users across various platforms. At NSPCC, we have always wanted to give web developers the full powers of NeoFS and its native API. However, to achieve this we had to face some challenges.

1. NeoFS API is a binary protocol with protobuf declaration, which is not quite suitable for web applications.
2. NeoFS API does not support a de facto standard mechanism of signing data in web applications using Wallet Connect.

After long discussions and experiments, NeoFS API started to support Wallet Connect signature scheme. The latest neofs-node versions already implement that. So, this opens all doors for the new NeoFS REST Gateway.

Try NeoFS REST Gateway

Important! This is the first version of REST Gateway and it contains many sharp edges and areas for improvement. We want to make it useful for all kinds of developers, so don’t hesitate to suggest features or leave feedback on issues page. Thank you!

REST Gateway is available as a binary, docker image, and a part of NeoFS All-In-One environment for local tests. To start, specify NeoFS Node endpoint (find public NeoFS endpoints at https://status.fs.neo.org). API will be available at http://127.0.0.1:8080/v1.

$ ./bin/neofs-rest-gw -p grpcs://st1.t5.fs.neo.org:8082
info neofs-rest-gw/config.go:195 no wallet path specified, creating ephemeral key automatically for this run
info neofs-rest-gw/config.go:379 added connection peer {"address": "grpcs://st1.t5.fs.neo.org:8082", "priority": 1, "weight": 1}
info metrics/service.go:33 service hasn't started since it's disabled {"service": "Pprof"}
info metrics/service.go:33 service hasn't started since it's disabled {"service": "Prometheus"}
Serving neofs rest gw at http://127.0.0.1:8080

REST Gateway uses Open API 2.0 specification, see spec/rest.yaml file. Find autogenerated docs at http://127.0.0.1:8080/v1/docs, and more NeoFS-specific documentation at http://127.0.0.1:8080/docs.

Why not NeoFS HTTP Gateway

Adding a control panel made us rethink the way the gateway works with requests, its parameters, and headers. While HTTP Gateway works with binary encoded structures, we wanted to provide a better experience for web developers with human-readable JSON structures in request bodies and responses.

To avoid mixing these approaches, we decided to keep HTTP Gateway as a simple, reliable, and blazingly fast data access point for NeoFS. This is the core of our incoming CDN service. You don’t have to choose one gateway implementation over another. It’s better to use both wisely.

REST Gateway application demo

Soon we are going to publish Panel.NeoFS — our web application to control NeoFS assets directly from your browser just with your wallet. It uses all features of REST Gateway and serves as a great example of the capabilities open for developers. Now look at the sneak peek of what is possible with the new REST Gateway

NeoSPCC continues to work on distributed decentralized object storage NeoFS. We have already launched the Testnet T5 with all our services.

Up-to-date information on the NeoFS network can now always be viewed on the new website https://status.fs.neo.org.

With the latest NeoFS release we have implemented a lot of improvements and updates to make the storage system more reliable, faster and safer.

Also, we will start soon a new service based on NeoFS — NeoFS.СDN. It will open the world of WEB3.0 for modern Web developers and will make it very easy to deploy your sites on the decentralized Internet. Stay tuned.

We will talk about a huge number of innovations in separate in-depth guides, but for now let’s look at basic operations through neofs-cli, so that you can try the NeoFS service without much preparation or deep technical knowledge.

How to start with NeoFS T5 Testnet

Create Wallet and get neofs-cli

First, you need to create a wallet and request N3 Testnet GAS tokens.

Get NeoGo (will be used here). NeoGo’s latest releases are available here. All wallet-related actions can be done with other wallet apps, here you can find more information.

Get neofs-cli binary from the latest release here.

$ wget https://github.com/nspcc-dev/neo-go/releases/download/v0.99.1/neo-go-linux-amd64 -O neo-go

$ chmod +x neo-go

$ cp neo-go /usr/local/bin/neo-go

$ neo-go --version
neo-go version 0.99.1

$ wget https://github.com/nspcc-dev/neofs-node/releases/download/v0.30.1/neofs-cli-amd64 -O neofs-cli

$ chmod +x neofs-cli

$ cp neofs-cli /usr/local/bin/neofs-cli

$ neofs-cli --version
NeoFS CLI
Version: v0.30.1 
GoVersion: go1.17.11

• Create an N3 wallet and check your address.

$ neo-go wallet init -w wallet.json -a

• To get your wallet address and the public key, you can use the following command:

$ neo-go wallet dump-keys -w wallet.json
NTRyEpPKFHaxj8uphYi54DJtbbCUtyYjyg (simple signature contract):
027810f6af30862884c7b0cd2d7705fffaa7e3254cecd8707fbd0223e8864c2be4

Now you can request N3 T5 Testnet GAS via the NGD N3 TestNet faucet.

1. Open NEO3 Testnet Faucet and request GAS.
2. Check your address here to see that everything is done correctly.

Make a Deposit

To make a deposit to the NeoFS account, execute the transfer method to the NeoFS Smart Contract address.

neo-go wallet nep17 transfer -w wallet.json -r https://rpc.t5.n3.nspcc.ru:20331 --from {address} --to  NZAUkYbJ1Cb2HrNmwZ1pg9xYHBhm2FgtKV--token GAS --amount {amount}

• {address} — N3 address from your wallet (NTRyEpPKFHaxj8uphYi54DJtbbCUtyYjyg in example above)
• {amount} — number of gas tokens to add to the NeoFS balance

example:

$ neo-go wallet nep17 transfer -w wallet.json -r https://rpc.t5.n3.nspcc.ru:20331 --from NTRyEpPKFHaxj8uphYi54DJtbbCUtyYjyg --to NZAUkYbJ1Cb2HrNmwZ1pg9xYHBhm2FgtKV --token GAS --amount 20

Enter password > 
Network fee: 0.00123252
System fee: 0.01451339
Total fee: 0.01574591
Relay transaction (y|N)> y
00fe1882fef5ab5edaca153b4cfb8949328873b4e52577b36341705a15b64ebd

Check NeoFS balance

To get NeoFS balance, execute the command below:

$ neofs-cli -r {NEOFS_ENDPOINT} -w wallet.json accounting balance

• {NEOFS_ENDPOINT} — any NeoFS node. Actual endpoints information is available here.

For example:

$ neofs-cli -r st1.t5.fs.neo.org:8080 -w wallet.json accounting balance 

Enter password > 
106.00173390

Create your first container

In NeoFS, users put their data into Containers. Containers are like folders in a file system or buckets in Amazon’s S3 but with Storage Policy attached. Storage Policy is set up by the user and defines how objects in this container should be stored.

The policy can use nodes attributes as follows, “Store data in three different countries on two different continents in three copies on nodes with SSD disks and good reputation.” Storage nodes will do their best to keep data in accordance with this policy. Otherwise, they will not get paid for their service.

To create a container, one has to set Storage Policy via neofs-cli command. For now, we will give a simple example of storing several copies of data on random storage nodes. For instance, to store an object in 3 copies, we should declare such policy as REP 3.

On T5 testnet, anyone can easily add and kill their own node, as this network is designed for testing and development. Therefore, we recommend for the first time to create a container with the storage rule 'REP 2 IN X CBF 2 SELECT 2 FROM F AS X FILTER "Deployed" EQ "NSPCC" AS F'. Such placement policy stores objects on nodes which are deployed by NeoSPCC.

There are several predefined basic Access Control Lists (ACL):

• private,
• public-read,
• public-read-write.

In a public-read-write container, everyone can read and write objects to the container. In a private container, only the owner of the container can execute read and write operations. In a public-read (read-only) container, only the owner can write to the container, but anyone can read data from it. In our example, we will use the predefined public-read basic ACL.

To do it, we should execute neofs-cli command as follows:

$ neofs-cli -r st1.t5.fs.neo.org:8080 -w wallet.json container create --policy 'REP 2 IN X CBF 2 SELECT 2 FROM F AS X FILTER "Deployed" EQ "NSPCC" AS F' --basic-acl public-read --await

Enter password > 
container ID: 7gHG4HB3BrpFcH9BN3KMZg6hEETx4mFP71nEoNXHFqrv
awaiting...
container has been persisted on sidechain

It can take some time to process the transaction on N3 Testnet.

Container ID for your data is 7gHG4HB3BrpFcH9BN3KMZg6hEETx4mFP71nEoNXHFqrv.

Once the container is created, we can upload data to the NeoFS network.

Upload a file to NeoFS

To put an object in our container, we should execute the neofs-cli command.

We can also add some user headers to the object to run search operations by specific filters in the future.

We will set content_tag as cat and my_attr as cute, for example.

$ neofs-cli -r st1.t5.fs.neo.org:8080 -w wallet.json object put --file cat.jpg --cid 7gHG4HB3BrpFcH9BN3KMZg6hEETx4mFP71nEoNXHFqrv --attributes content_tag=cat,my_attr=cute

Enter password > 
 233776 / 233776 [===============================================================================================================================================================================] 100.00% 1s
[cat.jpg] Object successfully stored
  ID: C4FWmxCGyv6vkXbnnAMXRFMSnkEfZCSG8FJ2WdEr2noS
  CID: 7gHG4HB3BrpFcH9BN3KMZg6hEETx4mFP71nEoNXHFqrv

And in addition, we will upload a video file cat.mp4.

Get your file or share it with friends

To get the object, you can use the command below with either any key (in case of public container) or an owner key (in case of private container):

$ neofs-cli -r st1.t5.fs.neo.org:8080 -w wallet.json object get --cid {CONTAINER_ID} --oid {OBJECT_ID} --file {PATH_TO_FILE}

Search for objects with some specific attributes in the container

To run search operation, you can use different filters by meta information of the objects. For example, you can filter objects by attributes declared in the previous steps:

$ neofs-cli -r st1.t5.fs.neo.org:8080 -w wallet.json object search --cid 7gHG4HB3BrpFcH9BN3KMZg6hEETx4mFP71nEoNXHFqrv --filters 'content_tag EQ cat' --filters 'my_attr EQ cute'

Enter password > 
Found 2 objects.
FVaR4RiwzeG79BURYHU4xVNE2BcHB7NaGpEDH5nsEGue
C4FWmxCGyv6vkXbnnAMXRFMSnkEfZCSG8FJ2WdEr2noS

Try our public HTTP Gateway

We provide a demo HTTP gateway, but recommend to deploy your own instance for your applications.

How to launch your HTTP-gateway — we will describe it in detail in the following articles.

In the meantime, you can use the description from the repository — https://github.com/nspcc-dev/neofs-http-gw.

To open a file in the browser just use links like:

https://http.t5.fs.neo.org/7gHG4HB3BrpFcH9BN3KMZg6hEETx4mFP71nEoNXHFqrv/{Object ID or file name}

For uploaded examples:

https://http.t5.fs.neo.org/7gHG4HB3BrpFcH9BN3KMZg6hEETx4mFP71nEoNXHFqrv/cat.jpg

https://http.t5.fs.neo.org/7gHG4HB3BrpFcH9BN3KMZg6hEETx4mFP71nEoNXHFqrv/cat.mp4

Disclaimer

Remember, though, that it is Testnet. If you have any problems, mail us via ops@nspcc.ru or file issues on Github. Also, be aware that depending on the capacity of the test network and production needs, we can periodically clear all data stored in N3 TestNet.

The service is designed for individuals 18 years of age or older. As a user of the service, you undertake to follow these terms of service and be responsible for all activities and content you post/upload. In addition to following these terms of service, you are responsible for adhering to all applicable local and national laws. NEO SPCC is not responsible for the content uploaded by users.

State components and options

The state stored in the DB can be roughly divided into several main categories:

• contract storage
• block headers
• block transactions
• MPT data
• transfer data
• auxiliary data (depends on implementation, but negligible in volume)

Contract storage

Contract storage is the essence of the system. While it grows over time, this growth reflects the use of the system and can’t be avoided. Its growth at the same time is quite modest and mostly associated with new accounts/contracts because shuffling the existing assets between old participants only changes the records in the DB but does not create them.

Block headers

Block headers are always stored in the DB, that’s where the block chain comes from, so they’re crucial. In Neo N3 a header is roughly 700 bytes, so for some 1–2M of blocks it’s not a lot (700–1400 MB), but of course, the number inevitably grows over the years of blockchain life.

Block transactions

What’s more interesting is transactions that are a part of a complete block. Neo N3 has two important protocol parameters related to them: MaxValidUntilBlockIncrement and MaxTraceableBlocks. MaxValidUntilBlockIncrement is tightly related to transaction’s ValidUntilBlock field that makes transaction incorrect after the height specified there; it specifies the maximum ValidUntilBlock value difference from the current blockchain height for a transaction to be considered correct. This is important for transaction validity checks, any new incoming transaction technically should be checked for duplicates only against the set of transactions from the previous MaxValidUntilBlockIncrement number of blocks, older ones just can’t have this transaction. On current Neo public networks (mainnet/testnet) this parameter is set to 5760, which is one day of 15-seconds blocks.

But this doesn’t mean any older transactions can be removed. When a transaction is run, it can access some data from the older blocks/transactions and the depth at which these requests can look is controlled by the MaxTraceableBlocks network setting. Neo Legacy never had that, it wasn’t possible with the UTXO model, but for N3 that’s one of the core settings. On current public networks it’s set to 2102400, which is one full year of blockchain life with 15-seconds blocks.

The key N3 protocol feature coming from the combination of these two parameters is that once a transaction becomes older than MaxValidUntilBlockIncrement and MaxTraceableBlocks relative to the current height, it can be safely deleted from the storage. Given the difference in defaults and the expected usage scenarios, usually it’s the MaxTraceableBlocks setting that is the most important regarding the ability to drop old data.

MPT data

Another and in fact very important source of data is Merkle-Patricia Trie that in some ways duplicates contract storage because it stores the same key-value set but in a radically different way. It allows computing a single hash value known as “state root” corresponding to a block. C# node implementation doesn’t have it by default, it’s implemented by the StateService plugin that needs to be installed, but NeoGo team believes that this data is so important (allowing to detect any state inconsistencies between nodes quickly) that MPT is always calculated and stored there without an option to turn it off (yes, it works even when we’re setting new performance records).

Given that any block does some state changes, MPT also changes with every block. Each key-value pair change affects all MPT nodes from the leaf changed to the top, for the current tries it’s roughly 10 nodes. This means that many new nodes get created with every block and if you don’t do anything, the DB grows very quickly. In fact, when this functionality was introduced into Neo Legacy (0.76.0 version on NeoGo timeline) the DB size exploded more than 2-fold, and very soon everyone realized that storing all of MPT data is quite expensive.

Removing any of this data is not easy, any trie for any block has some nodes of its own and also references a big set of nodes from previous block’s MPT, while some nodes can be reused even in the same MPT. Back then, the NeoGo team did some explorations around garbage collection schemes, but they performed very poorly, rendering the system hardly usable. The main problem was that we needed to do a full MPT traversal during the GC cycle and it can’t be done quickly. C# developers at the same time tried reference counting; and this approach turned out to be more viable, resulting in the implementation of KeepOnlyLatest option (NeoGo introduced it in version 0.78.2). This option makes the node store only one MPT (instead of the whole history of MPTs), in many ways like the node stores only the latest contract storage data. The DB size dropped back almost to where it had been before state tracking was implemented (but Legacy had UTXO, so many transactions never changed contract storage).

The same option is available in the current NeoGo versions, and its logic hasn’t changed, KeepOnlyLatest allows to drop all of old MPT data but the latest one. At the same time, additional logic was implemented in NeoGo 0.98.2, more on that below.

Transfer data

Tracking asset transfers is also one of the key things expected from most nodes. C# node provides that with TokensTracker plugin, which tends to be used a lot, while NeoGo has that built-in and always enabled. The log of transfers grows as transactions get accepted and processed, so we can’t ignore it too.

The evolution of RemoveUntraceableBlocks

The main option (not counting MPT-specific KeepOnlyLatest) to deal with old data was introduced in version 0.92.0 of NeoGo. It was named RemoveUntraceableBlocks, and it does exactly what you might expect of it — removes block data that is unreachable (as of MaxTraceableBlocks setting) from the current transactions.

As you might already have noted, this only allowed removing one part of the old DB data, but we also had the KeepOnlyLatest option to deal with MPT data, so the only thing left in this case was transfer data. While it was kept forever, it at the same time wasn’t very big in volume, at least not requiring a lot of attention for the chains we had back then, so this combination was sufficient in many ways.

Things have changed over time, the chains grew, but more importantly NeoGo has also introduced another extension in version 0.97.3 that allows synchronizing contract storage state over the P2P network (P2PStateExchangeExtensions option). This extension relies on MPT data and another NeoGo extension that puts state root hash into the block header (just to remind everyone — none of these can be used on current public networks). The problem was that this functionality requires keeping at least three MPTs at different heights, so it’s inherently incompatible with the KeepOnlyLatest setting. But, of course, we wanted to drop old MPT data from the storage while still using these extensions.

That spurred another round of experimentation, but the problem turned out to be tougher than originally expected. It was trivial to keep all MPT data, it was very easy to keep just one MPT, but keeping a set of MPTs was a real challenge. They couldn’t be reference-counted (that’d mean traversing the whole MPT for every block to increment all counters), and our previous garbage collection attempts failed to reach adequate performance.

It required us several months of trying various schemes and optimizations to finally find the way to do this. All required DB changes and logic were released in NeoGo version 0.98.2. The base of this approach is still garbage collection, but the changes made to the DB allowed us to avoid MPT traversals even for GC cycle. We’ve also implemented garbage collection for transfer data, so starting from version 0.98.2 NeoGo can now drop all obsolete data in any configuration.

Setting RemoveUntraceableBlocks now activates all of the logic responsible for dropping old data. There is also an associated configuration parameter, GarbageCollectionPeriod that defaults to 10000 and specifies how often (in blocks) garbage collection is performed.

Benchmark

What to expect from these options on public networks? That’s what we wanted to find out, so a test was set up using NeoGo 0.98.3 on two machines using block dumps from mainnet (1508926 blocks) and testnet (1599728 blocks). The test was performing offline synchronization from these dumps using the `db restore` command.

The two machines used are (both Linux-based):

• Core i7–8565U with 16 GB of memory
• Ryzen 9 5950X with 64 GB of memory

During our long quest for an optimal GC solution, we’ve also noticed that different databases react differently to variations in loads and schemes. Thus, we’ve performed the test on both databases NeoGo supports: LevelDB and BoltDB.

And we tested the following node configurations:

• default one (keeps everything)
• KeepOnlyLatest enabled (only one MPT stored, everything else untouched)
• RemoveUntraceableBlocks enabled (and default 2M+ MaxTraceableBlocks, for both networks this means that nothing is actually removed, but the DB is prepared for removal at 2M+ heights)
• MaxTraceableBlocks set to 100K blocks (and RemoveUntraceableBlocks enabled)
• MaxTraceableBlocks set to 10K blocks (and RemoveUntraceableBlocks enabled)

Warning! Changing MaxTraceableBlocks can make your node incompatible with the network, it’s a network-wide protocol parameter, not something to be changed by a node. Here we’re doing it just for the test; but even in this case it wasn’t possible for us to test 10K MaxTraceableBlocks setting for the testnet, some transaction there looks back 22K blocks, so it fails and then leads to a state that is incompatible with the subsequent transactions.

Mainnet results

• KeepOnlyLatest (KOL)
• RemoveUntraceable (RUB)
• MaxTraceableBlocks (MTB)

LevelDB/Core i7–8565U

BoltDB/Core i7–8565U

LevelDB/Ryzen 9 5950X

BoltDB/Ryzen 9 5950X

Testnet results

LevelDB/Core i7–8565U

BoltDB/Core i7–8565U

LevelDB/Ryzen 9 5950X

BoltDB/Ryzen 9 5950X

Comments

First of all, we can clearly see that all options work as expected with respect to the database size, removing all MPTs but one or keeping less and less of a tail makes the DB much more compact. The difference will only grow with time because the chain dumps we used don’t have even one full year of blockchain life.

At the same time, there is a clear performance overhead for garbage collection or reference counting which in many cases can be compensated by the effectiveness of the smaller DB. Enabling RemoveUntraceableBlocks with the network settings we have now clearly hits the performance in all cases, but if it was allowed to cut the tail more aggressively it could not only save disk space, but also save time.

Regarding the DB difference, LevelDB is still a good choice for archive nodes that want to keep everything, BoltDB tends to use more disk space and is slower when it comes to write-intensive loads. However, it performs significantly better in any tail-cutting scenario and we expect space-preserving options to be used a lot in production. Also it’s better for read-intensive loads in general and while what we’re doing here is just processing the chain, any real node will also serve a lot of other requests related to P2P/RPC activity and we expect these to be handled faster with BoltDB (like it works better in our TPS benchmarks).

Conclusion

As of today, RemoveUntraceableBlocks does not change much for public networks. This will definitely change as these chains grow, but private ones with more aggressive MaxTraceableBlocks settings already can win a lot by using it. And KeepOnlyLatest is still there and still is very effective (especially if used with BoltDB). So, all the options are there to slim down your node and they will be more and more relevant in future.

Multisigning in accounts and transactions

First off, we must notice that ”multisigning” can be referred to in two different contexts: one is about signatures for M-out-of-N keys accounts and another one is about a number of signers in a transaction (see our previous post on signers and scopes). These are quite different cases, we’ll use ”multisignature account” term for accounts corresponding to verification scripts with a number of keys and ”multisigned transactions” for transactions having more than one signer.

While the concept of multiple signers looks pretty obvious — we just add another signer to the array with appropriate witness; multisignature accounts are a bit more involved. In some ways they don’t differ much from ordinary ones (they’re still a 20- byte hash value), but their verification script has a number of keys, M and N parameters and a call to System.Crypto.CheckMultisig syscall (very similar to Legacy scripts once explained in an old article, just using new opcodes/interops). An example of a three-out-of-four verification script looks like this:

INDEX    OPCODE       PARAMETER                                                             
0        PUSH3                                                                              
1        PUSHDATA1    02103a7f7dd016558597f7960d27c516a4394fd968...    
36       PUSHDATA1    02a7bc55fe8684e0119768d104ba30795bdcc86619...    
71       PUSHDATA1    02b3622bf4017bdfe317c58aed5f4c753f206b7db8...    
106      PUSHDATA1    03d90c07df63e690ce77912e10ab51acc944b66860...    
141      PUSH4                                                                              
142      SYSCALL      System.Crypto.CheckMultisig (9ed0dc3a)

What it needs to be executed successfully is a corresponding invocation script that contains three signatures for transaction in question. Like this one:

INDEX    OPCODE       PARAMETER                                                                                                                           
0        PUSHDATA1    199bdd216a4c6c4548fa04c2e75a20cbcbcc85baecd...
66       PUSHDATA1    b17f98a9649938a035caf57dda7eb5d169f80620df2...
132      PUSHDATA1    6d5d1157541c033ce9ede5fe16893ca27b2154dd823...

Any signer account can be a multisignature account, and Neo allows to have a number of them used in a single transaction. In fact, multisignature accounts are used a lot in Neo, every mainnet block is signed by a five-out-of-seven consensus node multisignature account, committee can change network settings using its 11-out-of-21 account and oracle nodes sign oracle responses with three-out-of-four account. NeoFS uses multisignature accounts too for container or deposit management, both in mainnet and in sidechain.

The problem is that different keys used in multisignature account belong to different entities (otherwise it makes little sense using multisignature), and these entities need to cooperate to produce proper invocation script with all signatures needed. Historically, there had been various approaches to this problem for different purposes.

State of multisigning

Consensus and state validation

dBFT consensus is at the core of Neo, this is how Neo blocks get created and signed. Consensus nodes communicate over P2P network with messages packed into ExtensiblePayload. When a node sends a Commit message for proposed block, it includes a signature for this block made using node’s key. Messages are broadcasted through the network and eventually every other node gets enough commits to finish the block. Then, nodes can create an invocation script using these signatures and the block is ready (similar scheme is used by state validation nodes for their job).

What allows nodes to exchange signatures in this case is exactly this ExtensiblePayload which can be seen as a generalization over Neo Legacy’s ConsensusPayload. It’s one of the broadcasted payloads and there are not that many of them:

• blocks
• transactions
• extensible payloads

Every time we deal with a broadcasted message in untrusted environment the question is — what prevents the network from being DDoSed by these messages? Extensible payloads are limited in that they can only be signed by a well-known set of keys which includes consensus and state validation nodes. Joe Random can’t create valid extensible payload to harm the network. Nodes that are allowed to send extensible payloads can be considered semi-trusted, there are protections from any of them becoming rogue, but we’re not likely to see that happening.

We have a scheme here that works fine for a well-known subset of nodes (elected or controlled by committee) that create something special, like a block or a stateroot message. However extending it to a larger set of nodes is dangerous. They will compete for the same (relatively small) payload pool and the more nodes we add the more chances we get for some of them going wild. Allowing anyone to send this type of payload is just impossible.

Committee

We have a committee that from time to time does wonders, like reducing fees on mainnet; and it does them with transactions that use a special committee multisignature account. To collect appropriate number of signatures committee members use out of band communication channels (chats of various sorts), no one can see the transaction on the Neo P2P network until it’s signed. An incomplete (not fully signed) transaction is created by one of the committee members; then, he passes it to the next committee member who signs it and passes this new incomplete context to the next one. It all repeats until enough signatures are collected.

The context needs to be signed by the members one by one, which requires a lot of synchronization effort. For something that is not done frequently, something that is done manually and better yet something that doesn’t require a lot of signatures the process works; a group of unhurried people can use any messaging app of their choice to perform this. But it can’t be used for automated process: it’s not clear who is the first one to create a transaction, who’s the next one in turn to sign (and what if he is offline), synchronization channel itself might be unavailable, the system only works for some task- or application-specific scenario.

Oracle nodes

Oracle responses are created by a group of oracle nodes with (on current mainnet) three-out-of-four multisignature account. The process is completely automated of course, so it uses a quite different technique that at the same time has one thing in common with committee way of signing — it’s out of band, not using P2P network.

All oracle nodes have an RPC server exposed with submitoracleresponse API. And all oracle nodes have addresses of all other nodes in their configuration files, so they can directly connect to each other and exchange responses this way.

Obviously, this scheme doesn’t scale well, nodes always try to submit their data to each other, so it naturally leads to N ∗ (N − 1) connections created and messages exchanged between them for every transaction to be signed. It can also be hard to manage, moving any node to a different address requires updating the configuration of all the others. And the fact that every node directly knows each other node makes the approach only acceptable for app-specific cases.

Centralized collection

Oracle signature exchange scheme can be simplified by using some centralized entity that every node would connect to. But this would mean that either every application creates a service like that for its purposes, or someone has to run this generic service available for anybody. Running a service is not free both in terms of human resources and hardware cost, especially given the fact that this service would be a natural point of interest for attackers, so it’s not likely to happen as something available for everyone to use.

Even if someone to provide this service for general audience, it’d still be a single point of failure. Using it to sign a transaction for distributed decentralized application seems to contradict the essence of such application.

On-chain collection

Another natural approach used by many other blockchains is just managing signatures in a contract. This can also be done in Neo, and in fact NeoFS team has a lot of experience with this because that’s the way NeoFS contracts orignally worked. The idea is rather simple: contract keeps a set of authorized keys in its storage and then some method requiring M-out-of-N signatures is invoked by each signer in a separate transaction. Contract’s code checks if it has enough signatures on invocation. If that’s the case, it does the action required; if not it just increments the counter of signatures received and exits. In a pseudo-code it looks like this:

keys = getTrustedKeys() 
for k in keys { 
    if runtime.CheckWitness(k) { 
        signerKey = k break 
    } 
} 
if signerKey is null { 
    // return error 
}
storage.Put(requestID+signerKey, "") 
if countSigners(requestID) < m { 
    return // Wait for more signatures. 
} 
// Do the action.

Et voilà! Seems rather simple and even usable for trivial and rare occasions, but every method of every contract requiring multisignature approval needs code like this (and we need to be very careful to not mix signatures for different cases then), at least M transactions are needed to perform the task and (the most problematic) it’s impossible to precisely calculate system fee for these transactions.

Imagine four nodes trying to perform some action in a contract, each of them would send a transaction of its own that calls the method required. They can perform a test invocation and it will go ”wait for more signatures” way, spending some small amount of GAS in the process. But when all of these transactions get into the block (in unknown order!) one of them (crossing the ”M” threshold) will go ”do the action” route and if transaction only has system fee attached based on test invocation it’ll just run out of GAS and fail.

So a dApp performing such actions on-chain inevitably has to add some additional GAS to every transaction. The exact amount needed is unknown and it depends on the task performed, but it leads to huge GAS waste when actions are performed regularly. Even more importantly, it leads to potential instability of the dApp. Any update of any dApp contract or contract that it depends on can change the amount of additional GAS required, and it will easily break the dApp (the call will run out of GAS again instead of doing something useful).

That happened so many times to NeoFS that we just had no other choice but create something that works better.

Notary subsystem

Notary subsystem is a proposal for integration into future Neo releases. It’s implemented in NeoGo as an extension since version 0.93.0 (it’s even documented), stabilized and improved in various ways since then. The protocol is described in a GitHub issue and it consists of several key components:

• native contract
• P2P payload with a separate pool on every node
• node module (to be used by designated nodes)
• additional transaction attributes: Conflicts, Not Valid Before and Notary Assisted

This system uses P2P network to spread not-yet-fully-signed transactions, so a new broadcasted payload is added to do that. But as was noted above, broadcasting anything via P2P is a dangerous activity that needs to be limited somehow. The way notary payloads are limited is very reminiscent of the way regular transaction spreading is limited. Regular transactions are limited by the amount of GAS transaction sender has — when the sum of the fees for currently pooled transactions reaches account balance, no new transactions from this sender are accepted.

This mechanism couldn’t be used directly for notary payloads because they’re representing not-completely-signed transactions. Since they’re not yet valid, they can’t be accepted into the chain directly, and no one knows if they’ll be completed at all. So, unlike with normal transactions there is no guarantee that the fee will be written off the balance, which is crucial to ensure that eventually the pool will be depleted. Notary subsystem utilizes native contract for this purpose, to use the system user needs to send some GAS to the Notary contract first, which guarantees that there are funds to cover for transaction cost if needed.

Notary request itself then is a combination of two incomplete transactions in one payload: main one (intended to be completed) and fallback (to be sent if the main one doesn’t collect all signatures needed). Fallback transaction always has exactly two signers: Notary contract (which is a sender of this transaction) and request sender (that previously deposited some GAS to the Notary contract). Both transactions have Notary Assisted attribute that counts how many signatures are to be collected for this transaction; main one has whatever number it needs and fallback always has zero. This attribute is not free to use, transaction must attach additional fee for every collected signature plus one.

To make using Notary contract as a signer possible it also implements verify method. This method effectively is a one-out-of-N multisignature checker, so it accepts a parameter, one signature that needs to be made by one of the designated nodes. These nodes are the ones receiving the fee collected for using Notary Assisted attribute, similar to how fees are distributed among Oracle or Consensus nodes.

Fallback and main transactions are tightly coupled in the request: both have the same ”Valid until block” value, but fallback transaction also has ”Conflicts” attribute set to the hash of the main transaction and ”Not valid before” attribute set to some future block. ”Conflicts” attribute is a signal to consensus nodes that only one of the two conflicting transactions can be accepted. ”Not valid before” makes fallback transaction invalid until the chain reaches the specified block number; it’s like the opposite of ”Valid until block” field, very similar to X.509 ”Not valid before”/”Not valid after” pair

P2P notary payloads are being exchanged between nodes and all nodes maintain a pool of them (similar to transaction pool), but designated notary nodes (set by the committee in RoleManagement contract, similar to Oracle or State Validation nodes) additionally track payloads with the same main transaction. Parties that want to multisign their transaction can send payloads containing signatures independently; if and when notary node collects enough signatures to make the transaction valid, it’ll send it to the network. If for whatever reason this never happens, then after the chain reaches ”Not valid before” block from fallback transaction the same nodes will add a signature for the Notary contract signer and send fallback transaction into the network.

Fallback transaction is basically a hostage that guarantees payment for the service. Normally witnesses for the main transaction are assembled correctly and this transaction enters the chain (making fallback transaction forever invalid because of ”Conflicts” attribute). However, if the main transaction is not completed in time (and that time is determined by the ”Not valid before” attribute), then fallback transaction (or transactions coming from multiple sources) does that, ensuring that the main one will never be accepted, again because of the conflict. The GAS deposited to the Notary contract is only written off if fallback transaction is accepted, so for normal use depositing GAS once to the Notary contract should be enough.

While payloads are visible to everyone on the network and technically anyone can assemble a complete main transaction, notary nodes are motivated to do so with fees they get for providing this service. They at the same time can’t cheat by sending fallback transactions right away instead of main ones (because they’re ”Not valid before” some block), and if they have both valid fallbacks and main transaction, sending main transaction is more economically appealing because nodes will get more GAS for doing that.

The system is completetly decentralized and even one Notary node alive is sufficient for it to work. It at the same time is open for any dApp to use, there is no need to reimplement a lot of the same multisignature logic for every dApp, just some adjustments to the backend depending on application specifics. It’s much more reliable and economically appealing than on-chain collection, the number of transactions used is reduced at times.

While it was initially intended for NeoFS nodes that jointly manage a multisignature account and are able to produce the same transaction independently, it can also be used in a bit different fashion for multisigned transactions, adding another useful feature to the Neo stack.

Sponsored transactions

Unlike with out-of-band solutions, notary payloads can easily be observed on the network as they’re spreading through it. NeoGo has implemented an additional logic in its subscription mechanism for external applications to be able to get an event when a new payload is added into the pool. This then allows to react on such events: add some missing signature and make the transaction valid.

Because all signers are equal, sender’s signature can be added the same way. So a user can send a notary payload with multisigned transaction that consumes as much GAS as it needs to, but some other account (single- or multisignature or contract) pays for it. The backend of the application can then decide if this transaction should be sponsored or not. Unlike contract’s verify method, the backend doesn’t have any limitations in doing so, it can parse the script, it can test-execute it, it can check all the fields, whatever is needed (see GitHub issue also).

If there is something wrong with the transaction, the backend might just ignore this event, main transaction won’t be completed and fallback one will be sent. The number of these outstanding requests is also directly controlled by the amount of GAS deposited in the Notary contract. While accepting such requests directly poses a risk of DoS attack, Notary scheme ensures that sending a big number of unsatisfied requests will be paid for. So, applications can safely offer this service to their users (and it’d be much easier for them to do so by reusing this infrastructure).

Conclusion

Notary subsystem is still a proposal for the core protocol, but it’s already implemented in NeoGo and you can try it out. We’ve been using it successfully in NeoFS for more than nine months by now, and it greatly simplified both contracts and backend logic, making the system more reliable and performant.

Testing setup

We’re using neo-bench b0d9cb2c83fda5fef741e19df3497109754b8323, but this time we’ve decided to try more combinations of various parameters (more on that later) and stock NeoGo 0.98.0 on the same machine as for the recent tests (Ryzen 9 5950X, 64 GB RAM, SSD).

Standard mempool

First, we’ll repeat the same test as it was done with 0.97.2, using 50K mempool, 10 worker threads, and 1M transactions.

34800 TPS using LevelDB is already a 15% improvement over the latest result (if we’re to compare apples to apples), but BoltDB allows to push even more than that with an average of 38700 TPS. Back in the 0.97.2 test, we had not even shown the results for BoltDB because there was nothing interesting to see there, it was about the same as with LevelDB. But recent changes made to the node tend to favor this type of DB now, so it’s nice to have something to choose from.

As you can see, we still have a sawtooth-like pattern in this mode, and we know exactly what causes it since preview4 days. Even though we get better and better average results, the mempool size still limits what can be achieved — the node can’t put more transactions in a block than it has in the mempool and it can’t accept new transactions until current ones are completely persisted. Technically, the maximum number of transactions we can fit in a block is 65,535, so given this limitation a natural choice for a mempool size would be 131,070, that is two full blocks.

To ensure transactions come into the node at the maximum possible rate, we’ve also increased the number of threads pushing them via RPC from 10 to 30. We’ve used these setting previously; depending on the machine and node version, it can either allow to squeeze a bit more performance or just not be noticeable at all.

At the same time, we’ve noticed that even with 50K mempool, single-node test now finishes in less than 30 seconds. Thus we’ve decided to up the ante and push three times more transactions into the node: 3,000,000 of them.

Below are the results with all of these changes (that are mostly just proper node configuration for the test).

Tuned mempool

Single node

Proper node configuration changes the behavior radically and allows to reach 42400 TPS using LevelDB and 52100 with BoltDB. There is no tick-tock pattern anymore, and most of the time the system stays close to 60K TPS, but still there are some occasional dropouts that bring the average down.

The reason for this behavior is an increased block time; and that happens because of disk synchronization that takes more time than expected. NeoGo flushes changes to disk every second, most of the time it takes some milliseconds to do that, but depending on the DB and disk state, some writes can take a second or even more, that’s where the node can slow down (to recover quickly). That’s a clear area for future improvements for us, but still more than 50K average with 3M transactions looks like a good result.

One more thing should be noted about memory usage. Technically, it’s about the same as in the 0.97.2 version, but BoltDB shows much higher memory usage and the reason is that it maps whole DB file into memory. So while it can work with less than that available, for optimal performance it’s better to have enough physical RAM to fit all of it.

Four nodes

Our previous test clearly reached the settings limit, so just increasing the mempool wouldn’t be enough for this network. Hence, we’ve also decreased block time in this setup from five seconds to just two. While it seems very aggressive (four nodes need to agree on a quite big block during this timeframe), it actually works fine.

We can reach 12500 TPS using LevelDB here and 15400 TPS with BoltDB. It’s much more interesting to see how a number of nodes cope with this load and the behavior is more complex — blocks tend to vary both in size and time. Yet, the system deals with it and processes all 3M of transactions.

Conclusion

We should emphasize that while the system we’re running these tests on is quite capable, it’s probably not the top-notch hardware by the 2021 (soon to be 2022) standards. It lacks any fancy GPU, it’s not overclocked, and even the amount of RAM it has is somewhat unnecessary, it’s just that the machine has it, but all of these tests could easily fit into 16GB. So, properly configured N3 networks can deliver a lot of raw performance on commodity hardware, 50K TPS are possible for a single node and 15K for four nodes.

The question that is often being asked is “where is the limit”? And frankly, we don’t know yet. Although we’ve done most of the obvious optimizations and every additional percent costs more and more in terms of development and testing efforts, the Neo N3 protocol itself only stabilized relatively recently, so there are still some known areas for improvement. In any event, Neo N3 networks are there, they’re accepting more and more transactions every day, and we know that Neo will handle them.

Neo N3 transaction

First, let’s take a look at what exactly is a Neo N3 transaction.

We won’t dig into every field of it, they all of course have some purpose and meaning behind them, instead let’s concentrate on things that matter for our discussion:

• script is a NeoVM bytecode, also known as ”entry script”, this is what gets executed when transaction is processed by nodes
• signers is an array of accounts (Uint160 script hashes) with scopes
• witnesses is an array (with the number of elements exactly matching the number of signers) of script pairs containing verification script (that hashes to signer script hash) and invocation script (that usually contains signatures checked by verification script)

So transaction can be seen as a byte code container signed by a number of signers with each of them having some signature scope. This structure is quite different from many other blockchains, it doesn’t have a single sender or receiver (which also is the reason for ”from” parameter in NEP-17 ”transfer” method and other witness-requiring methods like ”vote” in NEO token) and every transaction actually bears some Turingcomplete bytecode with it. Combination of these factors makes Neo N3 transactions very powerful.

Whereas in other blockchains almost any non-trivial action might require deploying a contract, Neo allows to have any kind of logic in transaction itself (like calling multiple contracts or the same contract multiple times in a single transaction). And having multiple signers simplifies multi-party interactions a lot, again a single transaction with several signers can substitute a set of transactions with additional on-chain contracts used by other chains.

There is another degree of flexibility here in that signers are identified by script hashes and this gives a full power of NeoVM to every signer verification script. While this is not a unique feature of Neo, many newer blockchains tend to simplify this scheme down to trivial single-key signature checks justifying it by ”optimization”. That’s a bit different subject, but just keep in mind that NeoVM is there for you even in this case.

If we’re to look at transaction as some kind of task for blockchain to process, the difference between ”from-to-data” and ”script-signers” schemes can be summarized as the difference between ”tell the receiver that there is some data for him from this key” and ”execute some code signed by these guys with these scopes”. While the former makes for a nice messaging system (that for some strange reason does code executions as a side-effect for some messages), the latter is what can drive much more complex scenarios.

And still, developers working with smart contracts on Neo N3 platform know that if they’re to retrieve transaction from the Ledger contract or from the script container, they can clearly see a ”Sender” field with no scope data. So this of course needs an additional explanation.

Sender

Unlike most of Legacy transactions Neo N3 transactions are always paid for, there are no free ones and in a system where we legitimately can have a number of signers an obvious question is — who pays the bills? And this ”sender” notion is the answer to that. Structurally it’s just the first signer in a list of signers. It’s his GAS balance that the fees will be deducted from when block with transaction is to be processed.

Does the sender sign a transaction? Of course he does, there is no way around it, his valid signature is 100% there the same way any other signer’s signature. Does it mean that the sender witnesses and allows any action to be performed in any invoked contract in this transaction? No, and that’s the key.

Being a sender means paying the bills and nothing more; this implicit role can’t be compared to sender’s powers of other blockchains. In fact, Neo N3 allows one to be a sender and give no permissions at all, paying the fee but not allowing any actions requiring an account witness.

The other common misconception (probably based on experience with messagebased blockchains) is that ”Sender” is the same as ”Invoker” which is what happens when contracts call each other by passing messages. This is not the case with Neo, it has proper invocation stack in NeoVM and at any allowed depth level the sender you get for transaction is the same (as transaction is the same); retrieving the script hash of the calling contract is possible, but there is a specific interface for that (”GetCallingScriptHash”).

These are the reasons proper Neo N3 contracts shouldn’t be relying on sender account they can get for any kind of security purposes. Practically if contract looks at this field at all it’s very likely to be making some kind of mistake, because witnesses are to be checked with ”CheckWitness” call and sender hash doesn’t have a lot of use to it.

CheckWitness

So what exactly ”CheckWitness” does and what’s so special about it? Way back when in Neo Legacy it allowed to check a hash against a list of script hashes verifying this transaction. These hashes were taken from transaction inputs, attributes and other places depending on transaction type, so ”CheckWitness” basically unified all of these sources.

Neo N3 has much simpler transaction model, some of the old ”CheckWitness” logic is just not relevant, but in fact it has gotten even more powerful. The basic meaning of this function is the same, in general it answers the question of whether the account (script hash) approves (witnesses/verifies) modifying some state controlled by it. Usually it’s about asset movements, like NEP-17 transfer requires a witness from asset owner to perform transfer to another account. But different contracts can use it in different ways, for example NEO contract’s ”vote” method also requires an approval from voter even though it doesn’t change the balance of voter’s account. As accounts are identified by script hashes that’s what ”CheckWitness” accepts.

The first thing it does while performing the actual check is comparing this script hash to the script hash of the contract that has called the contract that is being currently executed. The presumption here is that when some contract calls another one it gives an automatic witness for this call because if it doesn’t want for the call to succeed it might as well not do it at all. This simple ”CheckWitness” logic allows contracts to do anything possible with regular accounts, this is what makes NeoBurger voting possible, for example.

Then ”CheckWitness” goes through signers list trying to find the requested hash. If it’s not there, there is no witness provided and the check fails. If the hash is present in the list then it compares witness scope for this signer against current execution context, if everything goes well it returns ”true”, but if the scope doesn’t fit the check will fail. Notice that it uses a whole list of signers with each of them having a scope, this logic can’t be replicated by a smart contract in any way, so correctly using ”CheckWitness” for approval is an important aspect of Neo N3 contract development.

Signature scopes

Here is a list of scopes that can be used by signers:

• None
• CalledByEntry
• CustomContracts
• CustomGroups
• Rules (since 3.1.0)
• Global

They can be combined (except for ”None” and ”Global”), and there are reasons and use cases for all of them. Let’s walk through the list, we’ll use single signer in our examples to simplify things, but having a number of them doesn’t change anything scope-wise.

Global

Let’s start exploring scopes with the simplest one — ”Global”. Some might argue that ”None” is even simpler, but that’s not exactly the case. We can say that ”Global” predates it because that’s the way Legacy chain witnesses worked, authorizing any action related to account in question in any contract invoked from a transaction. Sounds perfect, the transaction is signed, we know the signature is correct, and numerous other blockchains use similar model where this signature approves anything done to the account. And yet Neo N3 introduced a number of additional scopes, why is that?

The main reason is security and safety of transaction processing, especially in presence of malicious and/or broken contracts. If we’re to suppose that all contracts are perfect and the user only calls these perfect contracts from the entry script, then using ”Global” wouldn’t be a problem. But reality differs a bit from this ideal world.

There’ve been cases on Legacy network as well as on non-Neo chains where this interpretation of witnesses has led to loss of assets. For example, someone created a fake token with the same symbol and name as some legitimate one. This token technically is a contract which has a different hash from the original one, but then the creator of this token can airdrop arbitrary amounts of it to different addresses. Then users receiving this token can either confuse it with the original one or try to interact with it just out of curiosity. And that’s where the danger comes from — they’re invoking some seemingly benign method of this malicious token (like ”transfer”), but instead of transferring its own tokens the contract invokes NEO/GAS/any other token contract and transfers these tokens to itself or some other predefined address controlled by the malicious actor.

In this case with ”Global” signature scope (or complete lack of scoping) good contracts can’t differentiate legitimate calls from bad ones, it’s all ”transfer” calls in a transaction signed by the account the transfer is requested from. Whether it’s a direct call from the entry script or indirect one from the malicious contract doesn’t matter, there are cases where this behaviour (contract requesting transfer from user) can be valid and there are ones where it can’t. This scenario can be further extended if we’re to talk about multiple parties signing a single transaction, having approval for any of them makes such attacks even worse than they’re for a single account.

This original problem was one of the main triggers for signature scoping mechanism that Neo N3 has implemented. While ”Global” is still available for you to use, it’s not recommended, it’s hard to imagine any case that couldn’t be solved with more strict and safe scopes now. Let’s delve into these new possibilities.

None

A complete opposite of ”Global” is ”None”, it means that this signer forbids using his signature in any contract. Sounds counterproductive — why would he sign the transaction then? But in fact it’s very helpful when used for signers in sender role, with this scope they only pay the fees, but not do anything else.

It may seem an obscure case, but if you’ve used Oracle subsystem of Neo N3, you have also used ”None” witness scope. The oracle response transaction containing an answer to oracle request has two signers: the first one is the oracle native contract which holds GAS it accepted with an oracle request and the second one is oracle nodes multisignature contract (and also ”None” scope!). So we have oracle contract paying for the response transaction, but not allowing any other actions for this account and if you think about it for a while you may notice that this scheme wouldn’t be possible at all with ”Global” scope.

An oracle response transaction calls oracle contract first and then this contract invokes the callback specified when request was created. This callback in general can be anything, if there was no signature scoping someone could easily invoke ”transfer” method of GAS contract to get tokens belonging to oracle contract account and GAS couldn’t resist giving them to the caller, because oracle contract is one of the signers. But with ”None” scope ”CheckWitness” call in GAS contract fails, so it’s not possible (there are other oracle-specific reasons that make it impossible too, but that’s a bit different topic). So ”None” scope allows for safe transaction sponsoring in general, contract- or account-based, the sponsor can be sure that his assets are safe, and the only thing he does is paying fees with his GAS.

However, oracle response also contains another signer, also with ”None” scope. It may seem even more strange to have it, because it technically does nothing; ”CheckWitness” never works for it and it’s not involved in fee payments, and yet it’s an important part of the whole oracle scheme. That’s because an oracle response must be signed by oracle nodes, a specific set of keys stored in the ”RoleManagement” native contract. It’s a multisignature account requiring three out of four signatures for current public networks, even though it doesn’t normally have any assets an oracle response transaction shouldn’t allow messing with it. That’s also ensured by ”None” scope, the signature is there, it can be checked, but ”CheckWitness” will never allow doing anything with this account in such a transaction.

Simple as it is, ”None” is a very effective scope for such special cases.

CalledByEntry

This scope is a safe default covering most of the cases, that’s what you’ll be using most of the time when dealing with Neo N3. The permission is given to the entry script and ones it directly calls. Most of invocations do not require going deeper than that and even if some additional contracts are being called they might not require witnessing any actions (like calling ”StdLib” functions). An entry script is created by a user (OK, fancy UI controlled by a user), so user should trust it and it directly contains code that calls some other contract or contracts, so these calls are well known, user is likely to want them to happen and be authorized.

Notice how this simple restriction prevents the bad case described for ”Global” scope. If someone invokes bad contract with ”CalledByEntry” scope the permission would only be given to this contract itself. If it tries to steal user’s GAS/NEO/any other token that’d be at least one additional call down the invocation stack and ”CheckWitness” doesn’t work there for user’s account with ”CalledByEntry” scope.

In vast majority of the cases this scope is all you need. But there are more complex scenarios, ones where you have a set of interacting contracts requiring witnesses and that’s where other scopes might be handy.

CustomContracts

”CustomContracts” allows one to specify exactly the contract or set of contracts the witness is valid for. These allowed contracts are specified as script hashes and if ”CheckWitness” is called from one of them, it succeeds; if some other contract is to call it, it’ll fail.

It’s a nice addition to ”CalledByEntry” for cases with additional internal invocations from the contract called by the entry script, but they also need a witness. The user (frontend code) is supposed to know which contract is going to be called and add it to the list of witnessed contracts. If something goes wrong and an unexpected contract is called instead of the appropriate one the user is protected because it won’t have his witness.

CustomGroups

Very similar to ”CustomContracts” this scope allows to add witness for a group (or a number of groups) of contracts. Groups are a NEP-15 concept, they’re specified in contract’s manifest and, in general, a contract can belong to a number of groups. Each group is identified by a public key, and this association is confirmed by a signature verifiable using this public key (so Joe Random’s contract can’t be a part of the group unless he can forge a signature for it).

Grouping itself is not used very often, but there are cases where it’s very handy — a complex dApp can be split into a set of interacting contracts and then this set can be grouped using a single key to sign manifests. Transactions interacting with this dApp can then give witness to a whole group, and irrespective of how contracts call each other inside of the group everything will work smoothly.

Rules

It may seem that the set of scopes described above covers everything and in fact that was the set of scopes available at the time of the official Neo 3.0.0 release. But version 3.1.0 (and compatible neo-go 0.98.0) has introduced a new one: ”Rules”, and it has an interesting story behind it.

Notice that ”CustomContracts” and ”CustomGroups” scopes limit witness validity to some set of contracts or groups, but these contracts or groups may appear anywhere in the invocation stack. This leaves some potential for reentrance attack, especially given the NEP-11/NEP-17 ”onNEPXXPayment” functionality. Trusted (and witnessed) contract can call another contract which in turn can call trusted contract again, and it will get a valid witness even though it was never intended to.

There were various proposals to fix this, but the resulting mechanism we have now is ”Rules” scope, which technically can replace any of the above (but they of course are still available both for compatibility and simplicity reasons). ”Rules” scope is exactly that — a set of rules with predicates matching some condition and action performed if the match is successful. It can be seen as a set of firewall rules that are being examined one by one until some of them matches and returns a result.

The result, unlike with previous scopes, can be poth positive and negative, it’s also known as ”action”: ”allow” or ”deny”. If rule isn’t matched, it’s action is ignored, but to perform a match one writes a boolean expression using the following available conditions:

• Boolean: true or false, mostly useful for testing or emulating Global/None scopes
• Not: inverting nested condition
• And: matching a whole set of nested conditions (up to 16 of them)
• Or: matching one of conditions from nested set (also up to 16)
• ScriptHash: contains a hash to compare with script that is being currently executed (similar to CustomContracts)
• Group: contains a key identifying a group to compare with groups of the currently executing script (similar to CustomGroups)
• CalledByEntry: evaluates to true if the script is an entry script or one called directly by it (like CalledByEntry scope)
• CalledByContract: contains a hash to compare with the caller script hash
• CalledByGroup: contains a key identifying a group to compare with the groups of the caller script

This is not as simple as previous scopes, and yet it gives a lot of expression power to users and developers creating transactions. The case with reentrancy can now be easily solved by combining (And) checks for current group (Group) and calling group (CalledByGroup) or hash (CalledByContract) in a single rule. Notice that rules can be nested in some cases, but the depth of this nesting is limited to just two levels at the moment, which practically is enough given that you can add more rules (up to 16).

Conclusion

Scoped witnesses is one of the distinct core features of Neo N3. Deployed contracts can contain any code and invoke other contracts as they want, but with scoped signatures users now have control over what they approve to be done in their transactions and what not. Remember that the power of scopes is there, check witnesses correctly contractside, control what witnesses you give to whom on the user side — and the system will make your interactions much safer.

References

• Transaction details: https://docs.neo.org/docs/en-us/basic/concept/ transaction.html.
• Syscalls: https://docs.neo.org/docs/en-us/reference/scapi/interop.html.
• GH issue on voting from contract: https://github.com/neo-project/neo/issues/1924.
• CheckWitness documentation: https://docs.neo.org/docs/en-us/develop/write/basics.html#checkwitness.
• GH issue on scoped witnesses: https://github.com/neo-project/neo/issues/544.
• NEP-15: https://github.com/neo-project/proposals/blob/master/ nep-15.mediawiki.
• NEP-17: https://github.com/neo-project/proposals/blob/master/nep-17.mediawiki
• GH issue on Custom* scopes: https://github.com/neo-project/neo/issues/2583.

© NeoSPCC, CC BY-SA 4.0

Test setup

We do benchmarks using neo-bench, and it hasn’t changed much since the last post except for one small thing— we had to optimize the benchmark itself to reduce its own CPU consumption. That’s because we’ve got to the point where this tool started affecting the results (processing huge blocks is hard not just for nodes). Not in a big way, but still somewhat noticeable, so fixing it was important. Apart from that, it only received some adjustments regarding the protocol changes.

Hardware-wise, testing was performed using a Ryzen 9 5950X machine with 64GB of RAM and SSD. While on software side we’ve used the official Neo C# node 3.0.2 and NeoGo 0.97.2, both orchestrated by neo-bench revision number 09fe7c2bd587a12a44410e12674837be0c07523e. Both nodes used LevelDB as a storage backend.

Single node

Just a friendly reminder, our regular single-node test runs with one-second block interval and mempool capacity of 50,000 transactions. So, theoretically, the limit for this setup is 50K TPS, and Neo nodes are getting closer to it.

Both nodes have substantially improved their single-node performance with C# implementation providing an average of 7900 TPS and NeoGo 30300 TPS, that’s 48% and 55% more than what was observed in the preview4 benchmark respectively. There is a noticeable difference in how this was achieved though.

In preview4 release, C# node had experienced occasional drops in per-block TPS values due to lower number of transactions packed into a single block and inter-block time interval spiking up to 2 seconds and more. This is just not the case anymore, the node packs around 8000 transactions into every block during the whole test and delivers these blocks in time, deviating from the perfect one second by some mere 50–70 ms.

NeoGo node however still shows the tick-tock pattern on its graph for the same reasons as with preview4, but it has moved up somewhat and reduced the amplitude. More importantly, though it no longer shows any spikes in block times that could be seen with preview4, now it’s even closer to the target than C# node adding just 20–40 ms overhead. The line is somewhat shorter than for C# just because the benchmark runs out of transactions sooner (1,000,000 of them fit into 32 blocks).

Resource utilization just reflects the other plots, both nodes tend to produce more smooth lines there with NeoGo taking more cores for its job on average. Memory usage profile is somewhat more interesting though if we’re to compare it with preview4 one for NeoGo, because even though its TPS metrics have improved its memory utilization actually dropped at the same time by around 30%.

Four nodes

This network runs with a five second interval between blocks, so its theoretical limit is 10K TPS. And we’re getting really close.

NeoGo now reaches 8800 TPS on average and C# node delivers around 1000 TPS. C# node still experiences some problems with blocks containing 50K transactions (whole mempool), so the pattern is very similar to the one seen in preview4 and the 10% difference in numbers is actually less than what can be observed between successive runs of benchmark for this node.

But things radically changed for NeoGo node, once it warms up it just packs a little less than 50K transactions into every block until the end of the transaction stream and it does so with a typical block interval of 5100–5150 ms. Combination of these factors actually drives TPS metrics way up into the 9600–9700 range for the most of the test duration with average being somewhat lower just because of initial/ending blocks.

CPU utilization patterns are pretty much the same as were in preview4, but memory consumption has decreased substantially for both nodes.

Conclusion

As we can see, a single Neo N3 node can now provide about 30K (or 3W if you prefer) TPS in NeoGo implementation and 8.8K is no longer a big problem in networked scenario. While these raw numbers are measured in a somewhat sterile environment they still are important to understand where the limits are and what can be expected of a real network. Neo N3 protocol has a lot of potential in it and we’re ready to deliver this potential to our users.