Go(-lang) forums matters

New Oct2015, updated 15May2021.

This note is in groups Technology and My Go (golang) notes. I think it’s fair to copy my own comments at Go Forum in this note. I would also do newer golang-nuts and golang-dev matters that I have taken part in. The latter forums I won’t backlog.

You will have to go there to read the whole threads. These are basically my words.

Disclaimer: this blog not would basically be about matters I want to know more about!

Newest at the bottom:

Chans & goroutines vs Reactive Extension

(15Oct2015) I created this topic, see Chans & goroutines vs Reactive Extensions. My starting point for this was while writing an update in A reactive manifest. This again was triggered by a meeting I attended: Funksjonell reaktiv programmering i Swift, med Clojure backend (Trond H. Bordewich et al.) at www.digs.no here in Trondheim.

Interesting references in the Go Furum post are:

«I’d love to see a comparison of Golang’s goroutines/channels vs async»

https://news.ycombinator.com/item?id=9114064

GoFlow

https://github.com/trustmaster/goflow.
«GoFlow – Dataflow and Flow-based programming library for Go (golang).»

«This is quite a minimalistic implementation of Flow-based programming and several other concurrent models in Go programming language that aims at designing applications as graphs of components which react to data that flows through the graph.»

NSQ

https://github.com/nsqio/nsq.
«NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day.»

Further clicks:

http://nsq.io/overview/design.html.
«Regarding NSQ, Go channels (not to be confused with NSQ channels) and the language’s built in concurrency features are a perfect fit for the internal workings of nsqd. We leverage buffered channels to manage our in memory message queues and seamlessly write overflow to disk.» More here:

http://word.bitly.com/post/29550171827/go-go-gadget.
«Also, all of our C apps are built on libevent – i.e. a single-threaded event loop with callbacks that enables us to efficiently handle many thousands of simultaneous open connections. We use the same technique in Python via tornado, but the amount of work a single Python process can do is limited largely by runtime overhead. Additionally, this style of code can be hard to read and debug when you’re not used to it. Go’s lightweight concurrency model allows you to write in an imperative style while providing a built-in way to leverage the multi-core architecture of the computers we operate on.»

Some of my comments

Of course I understand the difference between a programming language and a pattern, between bricks and a house. It still puzzles me to see all these frameworks evolving from use with a non-concurrent languages (like Swift) that don’t have any notion of process (like gorotines) and communication (like channels) and synchronisation (like zero-buffered or full channels) ported and embraced on concurrent platforms (like go). Whether it’s push or pull with channels is kind of orthogonal to the problem, isn’t it? There can’t be any reception before production anyhow.

And it puzzles me to read things like the below (quote from ycombinator in my initial question):

(reubenbond at ycombinator) Channels in Go don’t compose: I can’t easily take one channel, mutate the values as they arrive, and create another channel from the mutated values. With Rx, that’s just var uppers = keyPresses.Select(key => key.ToUpper()) and now I’ve got a new stream of data. If keyPresses completes, so does uppers. If it fails, that failure is propagated through uppers. This isn’t easy in Go.
goroutines don’t have any reasonable kind of monitoring. I can’t say «this routine has completed/failed», I have to implement that notion every single time using a WaitGroup or something, and that only handles completion, not failure.

How correct are the answers? It’s not excactly the syntax «var uppers = keyPresses.Select(key => key.ToUpper())» on go, but it’s not much different, is it? Once channels have been connected between goroutines. «Composition: A Way to Make Proofs Harder» by Leslie Lamport certainly rules CSP to be compositional, and any goroutine (or PROC in occam) can spawn a new process. And with mobile channels (and go channels) you can spawn any channel you like at any time, except you’d have a way to connect that new channel (like sending it over an existing channel). Isn’t this enough?

Would a web server framework written in go for go from the onset look different than the others? It looks like GoFlow is one such? How does it differ from how a Reactive Extension for Go would have been?

(As you understand, I see this from an embedded programmer’s bench, but being curious about this)

Gorilla web toolkit

This is perhaps not a reactive framework as such, but then.. is it?

From https://groups.google.com/forum/#!topic/golang-nuts/m5U2Bkb88Dk I read «For session handling, roll your own or use Gorilla’s [1]»

This is perhaps what I’m after:  «roll your own or use Gorilla’s». Saying the same in a Swift or C++ world, I assume would be much more consequential. It’s not as easy to roll once own in those languages as in Go? So, those frameworks are mostly not driven by the Go society?

[1] «Package gorilla/sessions provides cookie and filesystem sessions and infrastructure for custom session backends», see http://www.gorillatoolkit.org/pkg/sessions

Broadcasting on a set of channels (golang-nuts)

See goolan-nuts (here). I was late in the discussion with this comment (here). The thread established the idiomatic way to do broadcast on a set of channels early (by Rob «Commander» Pike). Case closed…

But then the theme switched into sending pointers over channels, and it was said that this was ok. Hmm. I posted this question (less typos):

occam pi has/had mobile channels, where data pointed to by the pointer that’s implicitly sent over the channel will fall out of scope on the sender side. I don’t even think that data is converted to a const, the data won’t exist anymore. The language takes care of this.

Sending a pointer across without such a mechanism is rather risky, isn’t it – if one doesn’t introduce a «taken» return message and all parties respect it? If not, is it avoidable to insert data state and eventual polling?

A proposal for generic in go

See goolan-nuts (here). As always I am late (29June2016 at 15:42):

The suggestions of generics discussed here and in the referenced documentation, will it be possible to compile the «Go-with-generics» language into some idiomatic Go and compile it with the standard compiler? (I guess what idiomatic means is the real question here..). Or would it break idiomatic Go?

If it won’t break it, wouldn’t it be better to add some of this instead of leaving some of these important matters out? (I have read a guy mention concurrency and Haskell; the language is so flexible that it may be added any way. And it has. So there is no standard way for it)

Also, would it be possible to add generics around channels to (limit their functionality I assume (like not legal to address both sides of a channel in the same goroutine (I know it’s sometimes nice)))?

And make formal analysis (like transformation from Go-with-generics to CSPm) easier?

After some comments I added this:

I assume the scope of the discussion about introducing generics is not how Go-generics might hypothetically be transformed to idiomatic Go. I find no reference to «unsafe» in neither of these (already mentioned here):

Neither do I find any mention of «reflect», which I assume might be relevant.
From this I infer that adding generics to Go is a rather large affair. It also looks like that in the pages above.
More interesting, I don’t see formal modeling verification mentioned in any of the generics-documents above, neither in this recent thread:
I know that Rob Pike was involved with formal verification since long («Bell Labs and CSP Threads», by Russ Cox, see https://swtch.com/~rsc/thread/) and that this background was colouring much of how Go ended up. Has generics been discussed along this line: that Go-generics might be more/less suitable for formal verification?

There is an interesting comment after this.

golang-nuts: using channels to gather bundles from inputs for batch processing

This guy uses buffered channels for his problem. I posted a discussion on that theme, and mentioned some alternative patterns like the Overflow buffer pattern, the Knock-come pattern and the Go pattern shown in XCHAN – shown in the Appendix: Mixing Input and Output in the Go select Statement. Also, whether an output into a zero-buffered channel in a select with a timeout might do.

See https://groups.google.com/forum/m/?utm_source=digest&utm_medium=email#!topic/golang-nuts/lKJk90OCBxw (May2019)

Here are my references from that posting:

[1] Rob Pike: «Go concurrency patterns»: https://www.youtube.com/watch?v=f6kdp27TYZs&sns=em at Google I/O 2012. Discussed in [5]

Disclaimer: there are no ads, no gifts, no incoming anything with my blog notes, just fun and expenses:
[2] http://www.teigfam.net/oyvind/pub/pub_details.html#NoBlocking – See Figure 3
[3] https://oyvteig.blogspot.com/2009/03/009-knock-come-deadlock-free-pattern.html Knock-come
[4] http://www.teigfam.net/oyvind/pub/pub_details.html#XCHAN – XCHANs: Notes on a New Channel Type
[5] http://www.teigfam.net/oyvind/home/technology/072-pike-sutter-concurrency-vs-concurrency/

golang-nuts: concurrency safety

At golang-nuts (here) 26Nov2020. My comment the day after:

Thread safety, concurrency safety, goroutines safety. The same stuff to me.

It’s important that concurrent goroutines don’t interfere with each other in such a way that internal state is compromised by other goroutines. Typically through access to internal data, through pointers. If only go channels are used to communicate between goroutines, than that case is closed. That’s one of the reasons they exist – also in other similar languages, with that aspect often based on CSP.

But then, goroutines may communicate in a circle, the smallest would be two sending to each other and awaiting answer from each other at the «same» time. A system would easily freeze from this point and out. This would cause deadlock, which is pathological. Using deadlock free patterns may solve that problem from the beginning, provided such a pattern is used correctly.

Finally(?), one goroutine could be busy by itself, or by communicating with others, so that it is not able to read vital messages from other goroutines, ever. This would be a livelock.

I think go comes with a tool [1] that would help for some of these types of problems. I have not tested it myself, since I use another, embedded concurrent language. But I do read these mail lists every time they come, so I try to learn what this community is up to.

There would be ways whereby one could model concurrent programs as well. Lots of modelling languages exist. Myself I have to some extent used CSPm (with the FDR tool), FSP (with the LTSA tool) and Promela (with the Spin tool).

[1] https://golang.org/doc/articles/race_detector.html

There are some good comments also around my comment. Like this, by B Candler:

Here’s a video explaining how channels can replace various concurrency primitives: it’s worth getting your head around: https://www.youtube.com/watch?v=5zXAHh5tJqQ – GopherCon 2018: Bryan C. Mills – Rethinking Classical Concurrency Patterns. 

Some main points from Mill‘s excellent lecture:

  • An asynchronous API return to the caller before its result is ready
  • The problems with asynchronous callbacks (like in JavaScript) is described
  • Problems with futures and async/await pattern
  • In go: Asynchronous ≡ synchronous. In go, synchronous and asynchronous APIs are interchangeable
  • Concurrency is not asynchronousity
  • The downsides of condition variables (monitors)
  • A buffered channel can be used as a semaphore
  • Share data by communication the data
  • Share completion by completing communication
  • Events can be completions
  • Share a thing by communicating the thing
  • Worker pool pattern (=thread pool)
  • Start goroutines when you have concurrent work to do now

golang-nuts: Still «missing» priority or ordered select in go?

29Apr2021: See 049:[Still “missing” priority or ordered select in go?]

Some code created by people in the discussion thread (below). Observe that some of this code is not correct or runnable! Visit the thread to find out

9May2021: I «discovered» a blog note from 2012 that also takes up this theme, see Priority select in Go. So now it also is in the Technology list!

  1. https://go2goplay.golang.org/p/7xDzP6Jvyl8 (By Jan Mercl, 29Apr2021 11:15:14)
  2. https://go2goplay.golang.org/p/S_5WFkpqMP_H (By rog, 29Apr2021 23:52:05)
  3. https://go2goplay.golang.org/p/vhmo_Vw6OQy (By me, 2May2021 21:23)
  4. https://go2goplay.golang.org/p/EJ2J1sQXVEk (by me, 2May2021 21:32)
  5. https://go2goplay.golang.org/p/Min0kAI7Wao (xC example by me 3May2021)
  6. https://go2goplay.golang.org/p/ttr7sojJ-9T (Go with one case in each, by me 3May2021)
  7. https://play.golang.org/p/UUA7nRFdyJE (By Axel Wa.. 2May2021 20.23)
  8. https://play.golang.org/p/4G8CY36L0Qy (By Axel Wa.. 2May2021 20.23)
  9. Priority cases in select? – Old thread from 25Jan2017, started by tapi…@gmail.com
  10. An old problem: lack of priority select cases – Old thread from 28Aug2019, also started by tapi…@gmail.com
  11. There is No Now – Problems with simultaneity in distributed systems by Justin Sheehy – added by jesper.lou…@gmail.com on 6May2021
  12. https://play.golang.org/p/QfiPsh6udgz (By rog 6May2021)
  13. https://play.golang.org/p/LcWgDcX5ojl (By Axel Wa.. 2May2021)
  14. https://play.golang.org/p/B5W-UjKxlq_5 (by rog, 13May2021)

occam to go translation

On IEEE-COPA 2021 (COPA2021, COPA-2021)

Matilde Broløs, Carl-Johannes Johnsen and Kenneth Skovhede
Occam to Go translator
PRESENTER: Matilde Broløs
ABSTRACT. Occam is a programming language built on CSP, which has for many years been used for writing safety-critical systems used in space technology and at CERN among others. However, the language has not been developed or maintained for the last 25 years, which makes it difficult to maintain the programs which currently has a code base in Occam. As changing the entire code base for such systems will prove both expensive and time consuming, it is desirable to find an easy and secure way to translate Occam programs into another programming language. This paper lays the foundation of a transpiler from Occam to the newer programming language Go using Haskell. Go is a modern programming language which also implements many of the CSP principles found in Occam, making it a suitable target. The transpiler is implemented for a subset of Occam including only basic functionality, and is successful in translating simple programs from Occam to Go, showing that it is indeed possible to automatically translate Occam programs into Go.

See https://easychair.org/smart-program/COPA2021/