Lacking documentation

[KyleMaas]

Related to #19

For someone not already familiar with this project, I'm finding that the only way to actually figure out how anything does is to read through the code. The README is short and has some general directions to look. The Go docs page is helpful if you already know exactly what you want to do and just need to figure out the syntax for how. By far the most helpful thing has been https://dev.scuttlebutt.nz/#/golang/. But even then, most of the examples are how to use Margaret within the context of an Sbot. There's almost nothing about how to use Margaret directly, which makes it very difficult to try to debug issues within Sbot's usage of Margaret. The best resource I've found for that is reading through the test suite and trying to figure out what each test actually does. It would be really nice if there could be a few examples of how to use Margaret in isolation, such as:

1. Without an Sbot, how could you create a Margaret data store, store a message, and retrieve it again?
2. Without an Sbot, how do you configure indexes? How to they interact with the bitmaps?
3. Why does any user need to know whether it's using bitmap indexes internally or Badger ones? It's mentioned in the dev.scuttlebutt.nz page, but how does that difference affect users of Margaret directly?
4. Why is there so much very-nearly-but-not-quite-entirely duplicated code throughout the codebase? Even reading through it, I haven't been able to figure out the differences between implementations from what look to be different storage backends. (For example, do a search for the function "Query" and try and figure out which one Sbot's using in any particular context. And they all have subtle differences which don't seem to be documented anywhere.)
5. It would be great if there was an overarching conceptual description and diagram of how Margaret is structured, how the different pieces are intended to be used, how they interact, etc. Even a basic cursory overview would be helpful. If done properly and with enough detail, this could potentially also solve 3 and 4.

It's just...as a newcomer to this, my gosh the SSB code written in Go is tough to even start to grok. Very little is commented, and a lot of the comments are just descriptions of micro chunks of code that are fairly obvious in their function (i++ // this adds 1 to i kind of stuff) rather than an architectural description of how a module works as a whole and why it's done the way it is. If the barrier to entry could be lowered even a little bit, it would be a lot easier to attract development help. As it is, there's so much developmental decision-making history that gets taken for granted that everyone just knows that without some good documentation it takes reading through a substantial portion of the codebase to even get a general idea of how anything is constructed. For example, Margaret is described as Flume-like. I have never used Flume directly, so that is meaningless for me. And it's not even the same as Flume, so going and seeing how Flume works wouldn't even really help to figure out how Margaret works. The detective work to figure it out just feels like a grandiose waste of time that could be avoided.

And I'm sorry for the rant without a fix, but I'm not even qualified to be able to help fix it since I still don't even feel like I really understand the design concept, much less the details of the implementation.

In short, y'all are smart and know how this stuff works and the history behind its development. But not everyone who would like to help does. And we need your help to be able to help out in turn.

[KyleMaas]

As a good clear example of what I'm talking about, take a look at:

https://github.com/ssbc/margaret/blob/master/test/simple.go

This is a "simple" test. And yet there are nearly no comments throughout the entire thing, variable names are single letters, error messages are minimally descriptive, it uses interface{} all over the place, there are no explanations behind how the testing process is supposed to function, and it still doesn't even really say what it's supposed to be testing. It's all well and good if you already know how everything is supposed to function. It at least functions, but how? And yet the tests are one of the better references for how to use Margaret in isolation, which you really need to know when you're trying to work through a bug in Sbot in the vicinity of where it's using Margaret.

[decentral1se]

Thanks for the report @KyleMaas, this does seem like a specific focus and I certainly agree that docs could be improved.

To add a ref I've used and was shown to me via @cblgh, some helpful videos:

https://www.youtube.com/channel/UC4oSRf79DDEr7a2w5KO0gkw

Unsure how to unpack all of this rn but glad to have this open.