If Bitcoin wants to become a broadly accepted payment method, it will have to overcome some of its flaws that make it unattractive for many commerce applications. One of its biggest problems is “transaction speed”, which pretty much precludes it from day-to-day use and will limit its reach as a mean of payment. Crypto developers have taken on the challenge and developed new technologies that intend to improve or fix some of Bitcoins flaws. The Lightning Network is one of them and we wanted to give it a try.
We created a new Lightning Addon for Scipio ERP. It adds “dated currency conversion”, “historical prices”, the full integration of the LND mechanics and support of the entire order process, including invoicing, payment capturing, accounting and more. In short: it provides all the necessary steps to allow businesses to accept the new currency, while remaining in the legal framework government requires. If you already own a Lightning Wallet and a few Bitcoin, you can check out the implementation with one of our demo-products (if you want to receive order and payment confirmations, please provide a working email address).
Here is what we learned:
Lightning is a step in the right direction
I cannot overstate how essential transaction speed and reliability under load is for businesses. As a merchant, you’d want all order payments to be processed immediately and reliably so that the order process can continue. An interruption in the process can result in dissatisfaction for the customer, or a delay in your own workflow. Unfortunately, Bitcoin is too slow to process the payments and it takes an average of ~20 minutes until an order is “confirmed”. Any risk-averse merchant will wait for the confirmation before delivering digital goods or acknowledging the order of physical merchandise, therefore the process is slowed down. This is very problematic from an ergonomic point of view since customers are in the dark about the status of their order for an undetermined time span. At its worst, Bitcoin had spikes of > 11.000 minutes of confirmation time.
Lightning fixes this aspect by using its own network to collect and confirm transactions. And indeed, in our fastest tests, we managed to secure a confirmation within 5-10 seconds. That makes all the difference in terms of payment process and one should recognize the achievement by the Lightning community.
…but not without its own flaws
However, actual transaction speed is only one piece of the equation. As payment performance also needs to be reliable, merchants will want to also reduce dependence on single Lightning installations and therefore distribute their own traffic to multiple servers. Clustering, running several servers redundantly, is a cornerstone strategy for successful online businesses. The problem: Lightning does not really implement the means to allow for this kind of distributed setup. Lightning, or in this case the LND implementation, only allows incoming connections from a pre-configured list of IPs. In fact, the system is pretty much setup to run on the same server, i.e. allow the commerce application, bitcoin and lightning nodes to share the same server resources. There are ways to connect from an outside system, but setting it up is tricky and dynamic IPs won’t be an option. The competitor c-lightning shares a similar design. As a result, clustering, in particular in a dynamic cloud setup, is very hard and would require a lot of workarounds and configuration management.
Of course we should keep in mind that Lightning is still in its early stages, so I want to be less critical. But because of how Lightning currently is implemented, scaling the system will have its limits. Without the ability to scale, Lightning will remain unattractive for larger installations.
Lightning has a bad user experience
With most payment providers, the process of setting up and integrating them in you shop is rather painless: it takes a few minutes for an account to be created and a day or so for the bank account to be confirmed and off you go. Unfortunately, things are a little different in the Lightning world, or as this author summarizes his experience:
TL;DR: Sending payments using the Lightning Network is cheaper than the regular Bitcoin network, but suffers from routing errors and wallet bugs that make it impractical even for highly technical users.
Our experiences have been similar:
Interdependence of technology
Lightning is not exactly light-weight. You will require a Bitcoin node and a Lightning node and both should ideally be installed on the same server as your business application. All of which will requires a lot of time to setup and configure. Bitcoin will need ~10 days to synchronize the 250GB package (the blockchain) to operate. If one of the systems fails, your business application will not be able to handle cryptocurrency transactions. This is even more problematic, as redundacy cannot be easily achieved (see above).
Incredibly difficult to use
Lightning is not easy to use, neither for merchants or customers. Don’t believe me? Here’s the steps you have to go through to enable Lightning payments:
Merchant
- Setup Bitcoin node & wait for Bitcoin node to synchronize blockchain (~7-14 days expected wait time)
- Create a Bitcoin wallet
- Buy Bitcoin through Bitcoin exchange
- Fund Bitcoin wallet
- Setup Lightning node
- Create a Lightning wallet. For this you will need a cipher seed, ie. a password that has to be a 24 word string. It will look like this:
------------------BEGIN LND CIPHER SEED------------------ 1. one 2. two 3. three 4. four 5. five 6. six 7. seven 8. eight 9. nine 10. ten 11. eleven 12. twelve 13. thirteen 14. fourteen 15. fifteen 16. sixteen 17. seventeen 18. eighteen 19. nineteen 20. twenty 21. twentyone 22. twentytwo 23. twentythree 24. twentfour ------------------END LND CIPHER SEED--------------------There are generators for it – write it down!
- Unlock the wallet. This will generate a TLS certificate and key. If you want to access the wallet from an external system (and you managed to configure lightning to allow the access from that specific IP), you will have to make sure that your wallet is unlocked with a specific kind of certificate (x509 P-256 curve). The certificate Lightning creates is not of the right kind, so you will have to create it yourself with OpenSSL and override the original cert. Remember to restart lnd and unlock the wallet once the cert is changed and write down the cert location for later.
- Fund the wallet
- Open a Lightning channel
- Fund the channel, so that it can cover the transaction costs. The channel has to be funded by both parties. Since you are only interested in sending out invoices / payment requests to your customers, this would have to be your customer, although you can also take over the funding. The funding amount has to be sufficient (the rules of what is considered sufficient are not exactly straightforward, either).
- Wait for channel confirmation by at least 6 peers/broadcasters
- Remember the certificate and key? Copy them. The certificate and key will be required by each of your application systems, which will use it to connect to the node via client. This hard-binds the application server to a single lightning node.
- LND (a specific Lightning implementation) relies on “Macaroons”, a cookie-like file, which will be generated once you unlock your wallet. These have to be copied to you business application, too.
Remember: if you miss one step, you will not be able to accept payments.
Customer
- Buy Bitcoin through Bitcoin exchange
- Get Lightning Wallet or use the Lightning client
- Fund wallet from another address (Bitcoin wallet or otherwise)
- Open a new channel
- Fund your channel
- Go through checkout, receive Payment-Request. This is a string of ~228 characters length. Copy the string to your phone (or scan a QR-Code. Cross your fingers that the QR-Code is large enough, because 228 character strings don’t translate well to qr-code images).
- Accept payment and pay. By default you only have 30-60 Minutes to complete, otherwise Payment will terminate itself. (This is the default Lightning setting, although the merchant can modify this as part of his config)
- Wait for payment to be accepted by network
- Wait for payment to be processed by business application
Note: You can skip steps 1-6 on subsequent payments (unless you run out of Bitcoin).
Early-Stage technology
Lightning is fun, but very early-stage. We have been operating the system for 4 months and crashes can and will happen all the time. Transaction channels can close or may not have enough peers at any time. There are no push notifications for these events, so you won’t know until a new transaction is placed and fails. Just like with other online payment gateways – there are no offline payments to make up for this design.
Only small payments accepted
Lightning limits each channel to 0.16777216 Bitcoin. Once the limit is reached a new channel will have to be created. Likewise 0.04194304 Bitcoin are allowed for a single transaction. To bring this into perspective: as of writing a single channel would only allow payments in total of €500 and single transactions of €125. So if you are a merchant selling products through your online store, you’d only be allowed to create invoices of €125 or less and for every €500 of income a new channel would be required.
The €500 limit alone breaks the neck for many merchants. Meanwhile, the process of opening a channel and funding it with some Bitcoin value for the process fees will make it extremely difficult to automate for business applications.
So what’s the take away?
Lightning sets out to improve the overall performance of the Bitcoin-like technologies and at that it largely succeeds. It is faster in terms of transaction time and our tests have shown that, once set up, Lightning can process single transactions at a fraction of the time Bitcoin would.
Where Lightning fails is at its usefulness for professional clustered environments. Yes, Lightning does allow multiple nodes in its network, but at the same time it limits the number of systems that can connect to a Lightning node. And sure, payments are considerably faster than Bitcoin, but they are not “instant”. Even under best conditions, it still takes more time for transactions to be secured and processed than with you average credit card payment. In addition, Lightning is neither easy to setup, nor convenient to run.
All of which means that at the current state, Lightning payments, though interesting, are not ready to be used as the primary means for business transactions. That being said, the technology has potential and we will be updating our own components with future releases. Regardless, the demo we provided is fully functional and the component is available to Enterprise users and interested parties on request.
Thanks for this excellent and fair assessment of the lightning network. I am one of the people working on the protocol and getting this kind of feedback from relevant industries is extremely valuable.
Also I agree with the current issues and drawbacks that you mention. I believe however with the protocol update to BOLT1.1 and enhancements in implementations as well as upcoming 3rd party services most of the problems that you mention should vanish within 2019.
Would still love to chat with Paul on order to get some more insights. Maybe even on my YouTube channel on which I educate people about the Lightning network.
Hi Rene,
glad to be in touch. I was hoping that the feedback would be valuable to you guys.
I personally think that the biggest drawbacks are currently architectural decisions. Some can be easily fixed (transaction limits), others will be more difficult to tackle, but it is great that you are open for this discussion.
So obviously: yes, I’d love to chat. I will send you an email on the contact details.
> with the protocol update to BOLT1.1 and enhancements in implementations as well as upcoming 3rd party services most of the problems that you mention should vanish within 2019.
Most of the items listed in the post are purely documentation or software infrastructure issues. I don’t see any times that would be resolved by modifications to the protocol.
It seems to me the biggest problem at the moment is the slow rate of improvement in lightning infrastructure. How many years it would take for lightning to become usable, given the current rate of progress?
That’s an interesting point you are raising. We often discussed this internally as well.
Our hope is that the article addresses some of the problems, so that the community can get an idea of what we feel is missing the most. As you can see by the comment above – they seem like passionate guys, so I’m quite hopeful that they will manage.
> Clustering, running several servers redundantly, is a cornerstone strategy for successful online businesses
It’s trivial, and far more reliable, to set up multiple completely independent nodes and randomly distribute the traffic across them.
> Note: You can skip steps 1-3 on subsequent payments (unless you run out of Bitcoin)
You can skip to step 6!
You do not need a channel for every vendor.
Hi Tom and thanks for the feedback:
> It’s trivial, and far more reliable, to set up multiple completely independent nodes and randomly distribute the traffic across them.
Unfortunately, we haven’t been able to “break” the link between single lnd nodes and our application. Since you have to configure the allowed incoming IPs and also have to copy over the macaroons/certs, it seemed quite static to us.
> You can skip to step 6!
Fixed
Parts are factually incorrect.
It can take as little as 12 hours to sync a bitcoin node from scratch.
If it takes more than 24 hours… I’d recommend a different implementation, like BitcoinD. Lopp has done some investigating on sync speeds of different implementations.
“in our fastest tests, we managed to secure a confirmation within 5-10 seconds”
Sounds like you’re doing something wrong… and when combined with the “10 days to sync”… I’d say very wrong.
The longest I’ve waited for a LN payment is 3 seconds, and that was intentionally making the payment/route have to do retries.
“The channel has to be funded by both parties” Not only is this incorrect, it’s not even currently possible. [Though allowing this is on the agenda]
Opening and funding a channel are not two separate steps. You can’t open an unfunded channel. You currently can’t add funds to an existing channel… until splicing gets implemented.
“Wait for payment to be accepted by network” What does this even mean? There’s no context within the lightning network where “accepted by network” makes any sense.
“You only have 30-60 Minutes to complete” This is changable to much larger numbers, by the person receiving. Though, if you’re at a POS system standing in line 30-60 minutes before trying to pay… there are other issues at hand.
The limit is trivial to remove. It’s a single variable that you can set to whatever you want, before compiling the code. Though, it’d require the node partner to raise his limit as well. These limits were added on purpose, to reduce the #reckless from getting absurd. But with the drop in prices… they limits are way too low.
As for the rest, I generally agree with your concerns, but I’m confident these will get addressed over time.
Hey and thanks for the feedback.
You are right about the 30-60minutes. This can be modified by the merchant and I updated the text accordingly.
“Opening and funding a channel are not two separate steps.”
That should depend on whether or not the autopilot is on. We will check this again, but it is possible that we can simplify that statement.
“It can take as little as 12 hours to sync a bitcoin node from scratch.”
That largely depends on your hardware and the blockchain size when you synchronize the bitcoin node. Unfortunately, we haven’t found any other article pointing to much lesser synchronisation times. It made sense to us, as the blockchain size is increasing (https://www.statista.com/statistics/647523/worldwide-bitcoin-blockchain-size/), it will take more time to download and process the chain. That is why we stuck with our own assessment here. But even at 10 hours it would be too long for the kind of modern setup a business application would need. Just think about it like this: if a server crashes, you’d want to be able to power up a replacement as quickly as possible. Waiting for hours means loss of sales.
You can always backup the chain once it’s downloaded to upload it to a new server. That way you would only have to synch from the backup day. That should take only a few minute
> But even at 10 hours it would be too long for the kind of modern setup a business application would need. Just think about it like this: if a server crashes, you’d want to be able to power up a replacement as quickly as possible. Waiting for hours means loss of sales.
You can download a prepared blockchain from a already running node in your data center or even provision your server base images with a blockchain (would need monthly updates then). Also checkout the FAST-Sync that BTCPay-Server is using: https://github.com/btcpayserver/btcpayserver-docker/tree/master/contrib/FastSync
Yes, I think that would help.
Though generally speaking: I would prefer if Lightning wouldn’t rely on running your own Bitcoin node at all. In theory it should be possible to rely on a network of public bitcoin nodes that the LND connects to.
I understand that the reason for the low payment fees is probably the cause, but perhaps the architecture can change so that people who supply others with public nodes could receive a larger fee in return.
The payment fee is really low atm, so I don’t see why an increase would matter to merchants all that much and in return, it would drop the internal costs & risks of running and maintaining the required hardware significantly.
Hi Paul,
First thanks for much for this post! As a developer of lnd, feedback like this
is *extremely* useful. I think some of the items you mentioned here are possibly
due to a misunderstanding of the documentation of lnd, or perhaps unclear
documentation on our end.
> Lightning, or in this case the LND implementation, only allows incoming
> connections from a pre-configured list of IPs.
This isn’t the case. The TLS cert settings allow you to define either a domain,
or series of IP that we expect _incoming_ connections at. Alternatively, you can
use a proxy in front of `lnd` to bypass this and let something like nginx handle
the TLS termination. Some projects like BTCPay take this route.
> The problem: Lightning does not really implement the means to allow for this
> kind of distributed setup.
Can you elaborate on any other issues you ran into trying to make such a
distributed setup? Thanks!
> Lightning is not exactly light-weight. You will require a Bitcoin node and a
> Lightning node and both should ideally be installed on the same server as your
> business application.
This isn’t specific to Lightning, most merchants that are accepting larger
Bitcoin transactions _should_ run a full node for proper security. Full nodes
can also be run in a pruned mode which can utilize just a few GBs. Also 10 days
sounds very long, on modern hardware, most users are able to sync in less than a
single day.
lnd in particular also offers a “light client mode”, which will sync in minutes
(currently 1-2 mins on mainnet with an OK connection), and take up just a few
MBs at most. This is currently only available on testnet as we’re doing some
final safety and reliabilty work on it. However, it should be available sometime
early next year. At that point the documentation will be updated, and you’ll
also likely see it packaged as a major release. The rollout of this Lightning
light-client on mainnet should itself ease much of the on boarding burden.
> Lightning is not easy to use, neither for merchants or customers. Don’t
> believe me? Here’s the steps you have to go through to enable Lightning
> payments:
For sake of isolation, I think the steps should start with the user/merchants
already having Bitcoin. As that itself is an assumed starting state since the
Bitcoin on boarding process is a given here.
> if you want to access the wallet from an external system (and you managed to
> configure lightning to allow the access from that specific IP), you will have to
> make sure that your wallet is unlocked with a specific kind of certificate (x509
> P-256 curve). The certificate Lightning creates is not of the right kind, so you
> will have to create it yourself with OpenSSL and override the original cert.
This is incorrect. You don’t need to make a special cert in order to drive lnd
externally. You may need to set an external IP or domain (with –tlsextraip= or
–tlsextradomain=). Depending on what language you’re using to drive lnd
however, you may need to force it to accept our stricter set of cipher suites (no
sha1, no RSA, stuff like that). We have some documentation for this here:
https://github.com/lightningnetwork/lnd/blob/master/docs/grpc/python.md#imports-and-client.
The main thing to notice here is that you need to force gRPC to accept the
cipher suite:
“`
os.environ[“GRPC_SSL_CIPHER_SUITES”] = ‘HIGH+ECDSA’
“`
This can likely be better documented, thanks for bringing it to our attention!
> Fund the channel, so that it can cover the transaction costs. The channel has
> to be funded by both parties.
I think this section is a bit mistaken, it currently isn’t possible for nodes on
the network to both put funds into a channel. However, we’re working on some on
boarding tools to make the merchant on boarding experience to the network much
smoother.
> Remember the certificate and key? Copy them. The certificate and key will be
> required by each of your application systems, which will use it to connect to
> the node via client. This hard-binds the application server to a single
> lightning node.
This can be handled by any sort of automation or provisioning tools. Also the
application need not be bound to a single Lightning node. A load balancer can
be erected in front of the set of nodes to give the external application a
single endpoint they can hit to access the cluster of nodes.
> These have to be copied to you business application, too.
If you’re provisioning these nodes over RPC, then we’re working on a system to
allow a “stateless init”. In this mode, we don’t write the macaroon or certs to
disk, and will instead send it back over the RPC response. The program that’s
handling the provisioning can then hand off the authentication information to
any or instances that will need to contact the node.
> Cross your fingers that the QR-Code is large enough, because 228 character
> strings don’t translate well to qr-code images).
QR codes can easily handle up to 4k alphanumeric characters, perhaps this was an
issue in a particular QR code reader? Alternatively, an application can use
“deep linking” to communicate the payment information. This step isn’t unique to
Lightning, and applies to regular Bitcoin as well.
> We have been operating the system for 4 months and crashes can and will happen
> all the time.
If these haven’t been resolved, can you open issues on our repo to address them?
Thanks, also some issues may have been fixed in newer versions of lnd as well.
> Lightning limits each channel to 0.16777216 Bitcoin. Once the limit is reached
> a new channel will have to be created. Likewise 0.04194304 Bitcoin are allowed
> for a single transaction.
These are temporary limits that were put on as training wheels for the early
phase of the system, as it took some time for implementations to become more
mature. Upcoming versions will allow implementations to lift this limit.
Thanks again for such detailed feedback. If you ever need to get in contact with
our team directly, you can email me, or hop on our IRC channel (#lnd on
freenode) or developer Slack (you can find a link in the README for our repo)
— Olaoluwa
Oh, also there’ll soon be additional hooks for notifications when channels go up or, down or peers come online. How are y’all currently selecting channels atm?
That sounds great! It would improve the entire mechanism enormously.
Hey Olaoluwa,
so glad you got in touch and thanks for taking the time to respond. If you’re interested, we can have a brief exchange via email / skype, too.
That being said, perhaps I should respond a bit further on what you wrote:
>> This isn’t the case. The TLS cert settings allow you to define either a domain,
or series of IP that we expect _incoming_ connections at.
Yup, but that is precisely the criticism of ours. Systems like ours pretty much require a closed network to begin with, so even though you can secure the traffic, it isn’t really a huge concern. The preconfiguration of IPs means it is not really all that useful in terms of rolling it out in a dockerized environment and it also means that if you want to add a new system, you will have to stop, change the configuration and restart LND.
You can use a workaround like you described, in which you essentially mask the incoming traffic of all the commercial systems under a shared internal address. But that’s something merchants would have to come up with and it just adds to the complexity.
It would be easier to be able to turn-off the SSL in this context, or perhaps switch to simple self-signed SSL certs so that incoming clients can always connect to the node.
>> This isn’t specific to Lightning, most merchants that are accepting larger Bitcoin transactions _should_ run a full node for proper security.
You are right, it is more a cryptocurrency thing. But that doesn’t make it attractive for merchants to accept. Don’t forget: our clients will compare Lightning to existing payment gateways, if it can’t compete for the reasons listed, they won’t adopt it.
One thing I often wondered in the past months is: why doesn’t lightning provide public nodes businesses can connect to and use macaroons as way to secure the transactions. That way people can provide the infrastructure needed for businesses to operate and businesses will have an easier way to adopt the technology.
>> lnd in particular also offers a “light client mode”, which will sync in minutes (currently 1-2 mins on mainnet with an OK connection), and take up just a few MBs at most.
Is that only the lnd bit, or also the Bitcoin node? Because we really see Lightning as a package deal with Bitcoin atm (and yes, I am aware that you can operate lightning with other cryptocurrencies).
>> For sake of isolation, I think the steps should start with the user/merchants already having Bitcoin. As that itself is an assumed starting state since the Bitcoin on boarding process is a given here.
Our perspective is a business-driven one. If LND requires Bitcoin, we must list the steps accordingly as it will be required by businesses and customers for it to work.
>> You may need to set an external IP or domain (with –tlsextraip= or –tlsextradomain=). […]
That’s what we meant with ” you will have to create your own cert”. We followed this explanation here: https://github.com/mably/lncli-web/blob/master/README.md, which asked us to do the following:
cp tls.key tls.key.bak
openssl ecparam -genkey -name prime256v1 -out tls.key
openssl req -new -sha256 -key tls.key -out csr.csr -subj ‘/CN=localhost/O=lnd’
openssl req -x509 -sha256 -days 36500 -key tls.key -in csr.csr -out tls.cert
lnd –bitcoin.active –debuglevel=debug –bitcoin.node=bitcoind –bitcoind.rpcuser=ilscipio –bitcoind.rpcpass=… –bitcoin.mainnet –bitcoind.zmqpath=tcp://127.0.0.1:28332 >> /home/bitcoin/.lnd$
rm csr.csr
lncli unlock
openssl x509 -in tls.cert -text
It wasn’t until we did the above, that the setup would work. Are we missing some info here?
>> If you’re provisioning these nodes over RPC, then we’re working on a system to allow a “stateless init”.
Agreed – We’ve been following the progress on that. That would be an enormous improvement.
>> QR codes can easily handle up to 4k alphanumeric characters, perhaps this was an issue in a particular QR code reader?
To my knowledge, it is more a matter of the available size (in pixel) of the QR code itself, and the error correction level you are using. The error correction level will reduce the available characters and so will the QR code size itself. In our case: we started with a 100x100px, which was simply too tiny for the information. The jpg would smudge and the reader wouldn’t work, it was only after increasing the QR code size, the reader would work.
The point here is: the strings are enormous and handling them makes you look for tools like a QR code reader, so that people don’t have to type the strings. But since they are so enormous, they can still be problematic.
Using deep-links is a solution for it, agreed. Perhaps LND can provide shortener function for this and store a a shortened PR in the invoice info?
>> […]4 months and crashes can and will happen[…] > If these haven’t been resolved, can you open issues on our repo to address them?
We have been working with the Go releases of LND and pull regularly, so it should be fairly recent. I will ask my colleague, Martin, to open up issues on Github for you guys.
>> These are temporary limits that were put on as training wheels for the early phase of the system
Understood – we never saw it as a permanent restriction, but one that makes it unusable for serious merchants… the hard limit on channels is a nightmare in particular, btw.